Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 07:58:20.788182 398 test_util.cc:268] Using random seed: 357533118
I20240503 07:58:20.795318 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:58:20.795387 398 ts_itest-base.cc:113] --------------
I20240503 07:58:20.795403 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 07:58:20.795420 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:58:20.795433 398 ts_itest-base.cc:116] --------------
2024-05-03T07:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:58:20Z Disabled control of system clock
I20240503 07:58:20.805148 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43995
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:43995 with env {}
W20240503 07:58:20.922879 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:20.923137 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:20.923178 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:20.926302 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:58:20.926412 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:20.926445 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:58:20.926472 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:58:20.930410 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:43995
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:20.931365 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:20.932559 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:20.940654 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:20.941076 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:20.941670 410 server_base.cc:1034] running on GCE node
W20240503 07:58:20.942056 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:20.942667 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:20.943706 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:20.944852 410 hybrid_clock.cc:648] HybridClock initialized: now 1714723100944818 us; error 51 us; skew 500 ppm
I20240503 07:58:20.946960 410 webserver.cc:460] Webserver started at http://127.0.99.190:39027/ using document root <none> and password file <none>
I20240503 07:58:20.947562 410 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:20.947623 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:20.947813 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:20.949611 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d2b56ca4d0294343adcdcb049f20e45b"
format_stamp: "Formatted at 2024-05-03 07:58:20 on dist-test-slave-bj8b"
I20240503 07:58:20.950101 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d2b56ca4d0294343adcdcb049f20e45b"
format_stamp: "Formatted at 2024-05-03 07:58:20 on dist-test-slave-bj8b"
I20240503 07:58:20.953668 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 07:58:20.956087 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:20.957161 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:20.957280 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d2b56ca4d0294343adcdcb049f20e45b"
format_stamp: "Formatted at 2024-05-03 07:58:20 on dist-test-slave-bj8b"
I20240503 07:58:20.957376 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:20.976117 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:20.976878 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:20.977073 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:20.983954 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:43995
I20240503 07:58:20.983970 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:43995 every 8 connection(s)
I20240503 07:58:20.985072 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:58:20.988031 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:20.991379 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 410
I20240503 07:58:20.991509 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:58:20.993558 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b: Bootstrap starting.
I20240503 07:58:20.996372 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:20.997133 477 log.cc:824] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:20.999176 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b: No bootstrap required, opened a new log
I20240503 07:58:21.001799 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b56ca4d0294343adcdcb049f20e45b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43995 } }
I20240503 07:58:21.002010 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:21.002094 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b56ca4d0294343adcdcb049f20e45b, State: Initialized, Role: FOLLOWER
I20240503 07:58:21.002558 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b56ca4d0294343adcdcb049f20e45b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43995 } }
I20240503 07:58:21.002686 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:21.002748 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:21.002849 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:21.003767 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b56ca4d0294343adcdcb049f20e45b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43995 } }
I20240503 07:58:21.004107 477 leader_election.cc:302] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2b56ca4d0294343adcdcb049f20e45b; no voters:
I20240503 07:58:21.004395 477 leader_election.cc:288] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:21.004804 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:21.005030 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [term 1 LEADER]: Becoming Leader. State: Replica: d2b56ca4d0294343adcdcb049f20e45b, State: Running, Role: LEADER
I20240503 07:58:21.005367 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b56ca4d0294343adcdcb049f20e45b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43995 } }
I20240503 07:58:21.005678 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:58:21.006587 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2b56ca4d0294343adcdcb049f20e45b. Latest consensus state: current_term: 1 leader_uuid: "d2b56ca4d0294343adcdcb049f20e45b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b56ca4d0294343adcdcb049f20e45b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43995 } } }
I20240503 07:58:21.006700 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:21.008253 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2b56ca4d0294343adcdcb049f20e45b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b56ca4d0294343adcdcb049f20e45b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43995 } } }
I20240503 07:58:21.008499 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b [sys.catalog]: This master's current role is: LEADER
W20240503 07:58:21.010696 497 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 07:58:21.010776 497 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 07:58:21.010831 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:58:21.011715 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:58:21.016633 489 catalog_manager.cc:1347] Generated new cluster ID: 90c71e365ecd48438cf4f9dc0ccfe94a
I20240503 07:58:21.016706 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:58:21.059525 489 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:58:21.060554 489 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:58:21.066648 489 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d2b56ca4d0294343adcdcb049f20e45b: Generated new TSK 0
I20240503 07:58:21.067740 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:58:21.075937 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43995
--builtin_ntp_servers=127.0.99.148:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 07:58:21.189358 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:21.189626 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:21.189674 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:21.192992 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:21.193130 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:58:21.197111 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43995
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:21.198151 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:21.199416 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:21.208336 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:21.208727 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:21.209234 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:21.209750 501 server_base.cc:1034] running on GCE node
I20240503 07:58:21.210175 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:21.210844 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:21.212000 501 hybrid_clock.cc:648] HybridClock initialized: now 1714723101211985 us; error 36 us; skew 500 ppm
I20240503 07:58:21.213954 501 webserver.cc:460] Webserver started at http://127.0.99.129:41725/ using document root <none> and password file <none>
I20240503 07:58:21.214612 501 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:21.214676 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:21.214859 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:21.216673 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "99ada3d57094474183654dc5e16d5d44"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.217166 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "99ada3d57094474183654dc5e16d5d44"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.220754 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:58:21.223165 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.224242 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 07:58:21.224361 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "99ada3d57094474183654dc5e16d5d44"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.224465 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:21.235996 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:21.236733 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:21.236897 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:21.237469 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:21.238509 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:21.238559 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.238616 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:21.238641 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.247328 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:42775
I20240503 07:58:21.247349 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:42775 every 8 connection(s)
I20240503 07:58:21.248368 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:58:21.251724 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 501
I20240503 07:58:21.251818 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:58:21.254076 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43995
--builtin_ntp_servers=127.0.99.148:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 07:58:21.258203 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43995
I20240503 07:58:21.258482 629 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:21.259049 629 heartbeater.cc:511] Master 127.0.99.190:43995 requested a full tablet report, sending...
I20240503 07:58:21.260807 441 ts_manager.cc:194] Registered new tserver with Master: 99ada3d57094474183654dc5e16d5d44 (127.0.99.129:42775)
I20240503 07:58:21.262560 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:55039
W20240503 07:58:21.369254 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:21.369498 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:21.369541 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:21.372709 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:21.372838 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:58:21.376706 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43995
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:21.377724 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:21.378962 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:21.388114 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:21.388566 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:21.389024 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:21.389515 632 server_base.cc:1034] running on GCE node
I20240503 07:58:21.389907 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:21.390528 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:21.391678 632 hybrid_clock.cc:648] HybridClock initialized: now 1714723101391665 us; error 31 us; skew 500 ppm
I20240503 07:58:21.393596 632 webserver.cc:460] Webserver started at http://127.0.99.130:38545/ using document root <none> and password file <none>
I20240503 07:58:21.394186 632 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:21.394246 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:21.394490 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:21.396289 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d7ff088811de40ebb8baa70d2db6195f"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.396800 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d7ff088811de40ebb8baa70d2db6195f"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.400369 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:58:21.402668 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.403770 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 07:58:21.403880 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d7ff088811de40ebb8baa70d2db6195f"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.403972 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:21.423318 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:21.424083 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:21.424266 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:21.424890 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:21.426399 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:21.426450 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.426512 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:21.426537 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.436549 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:45277
I20240503 07:58:21.436563 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:45277 every 8 connection(s)
I20240503 07:58:21.437610 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:58:21.440145 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 632
I20240503 07:58:21.440268 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:58:21.442765 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43995
--builtin_ntp_servers=127.0.99.148:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 07:58:21.447952 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43995
I20240503 07:58:21.448254 760 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:21.449082 760 heartbeater.cc:511] Master 127.0.99.190:43995 requested a full tablet report, sending...
I20240503 07:58:21.450472 441 ts_manager.cc:194] Registered new tserver with Master: d7ff088811de40ebb8baa70d2db6195f (127.0.99.130:45277)
I20240503 07:58:21.451287 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:56283
W20240503 07:58:21.563526 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:21.563772 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:21.563817 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:21.567059 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:21.567183 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:58:21.571080 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43995
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:21.572084 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:21.573274 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:21.582056 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:21.582496 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:21.583027 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:21.583541 763 server_base.cc:1034] running on GCE node
I20240503 07:58:21.583961 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:21.584596 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:21.585762 763 hybrid_clock.cc:648] HybridClock initialized: now 1714723101585742 us; error 36 us; skew 500 ppm
I20240503 07:58:21.587808 763 webserver.cc:460] Webserver started at http://127.0.99.131:43369/ using document root <none> and password file <none>
I20240503 07:58:21.588477 763 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:21.588539 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:21.588725 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:21.590641 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e6b2fee74eb046268f08c397f8885bf7"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.591131 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e6b2fee74eb046268f08c397f8885bf7"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.594743 763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:21.597121 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.598233 763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:21.598381 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e6b2fee74eb046268f08c397f8885bf7"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.598496 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:21.615015 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:21.615741 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:21.615906 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:21.616501 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:21.617472 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:21.617522 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.617575 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:21.617600 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.627739 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:39489
I20240503 07:58:21.627759 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:39489 every 8 connection(s)
I20240503 07:58:21.628839 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:58:21.629036 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 763
I20240503 07:58:21.629128 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:58:21.631467 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43995
--builtin_ntp_servers=127.0.99.148:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 07:58:21.638557 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43995
I20240503 07:58:21.638854 891 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:21.639621 891 heartbeater.cc:511] Master 127.0.99.190:43995 requested a full tablet report, sending...
I20240503 07:58:21.640743 441 ts_manager.cc:194] Registered new tserver with Master: e6b2fee74eb046268f08c397f8885bf7 (127.0.99.131:39489)
I20240503 07:58:21.641327 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:49685
W20240503 07:58:21.747192 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:21.747443 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:21.747496 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:21.750707 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:21.750845 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:58:21.754763 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43995
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:21.755789 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:21.757050 894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:21.765899 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:21.766301 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:21.766824 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:21.767319 894 server_base.cc:1034] running on GCE node
I20240503 07:58:21.767772 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:21.768381 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:21.769528 894 hybrid_clock.cc:648] HybridClock initialized: now 1714723101769511 us; error 35 us; skew 500 ppm
I20240503 07:58:21.771447 894 webserver.cc:460] Webserver started at http://127.0.99.132:43505/ using document root <none> and password file <none>
I20240503 07:58:21.772053 894 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:21.772116 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:21.772294 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:21.774096 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "27074610401e48ca898230872e848b7f"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.774652 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "27074610401e48ca898230872e848b7f"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.778096 894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:21.780565 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.781632 894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 07:58:21.781742 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "27074610401e48ca898230872e848b7f"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.781831 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:21.805780 894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:21.806633 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:21.806800 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:21.807411 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:21.808393 894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:21.808442 894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.808503 894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:21.808528 894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.818197 894 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:44843
I20240503 07:58:21.818221 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:44843 every 8 connection(s)
I20240503 07:58:21.819270 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:58:21.827584 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 894
I20240503 07:58:21.827706 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:58:21.828410 1022 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43995
I20240503 07:58:21.828677 1022 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:21.829334 1022 heartbeater.cc:511] Master 127.0.99.190:43995 requested a full tablet report, sending...
I20240503 07:58:21.829977 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43995
--builtin_ntp_servers=127.0.99.148:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 07:58:21.830376 441 ts_manager.cc:194] Registered new tserver with Master: 27074610401e48ca898230872e848b7f (127.0.99.132:44843)
I20240503 07:58:21.830965 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:41957
W20240503 07:58:21.947472 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:21.947737 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:21.947784 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:21.951133 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:21.951263 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:58:21.955415 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43995
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:21.956548 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:21.957854 1025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:21.966982 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:21.967505 1025 server_base.cc:1034] running on GCE node
W20240503 07:58:21.967845 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:21.968327 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:21.968809 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:21.969486 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:21.970645 1025 hybrid_clock.cc:648] HybridClock initialized: now 1714723101970626 us; error 36 us; skew 500 ppm
I20240503 07:58:21.972709 1025 webserver.cc:460] Webserver started at http://127.0.99.133:35561/ using document root <none> and password file <none>
I20240503 07:58:21.973318 1025 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:21.973381 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:21.973603 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:21.975556 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0a385716c25a4f9d99f9aa81cd521d36"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.976087 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0a385716c25a4f9d99f9aa81cd521d36"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.979874 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 07:58:21.982264 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:21.983480 1025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:21.983608 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0a385716c25a4f9d99f9aa81cd521d36"
format_stamp: "Formatted at 2024-05-03 07:58:21 on dist-test-slave-bj8b"
I20240503 07:58:21.983719 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:22.023434 1025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:22.024226 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:22.024413 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:22.025045 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:22.026078 1025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:22.026129 1025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:22.026185 1025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:22.026209 1025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:22.036155 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:41945 every 8 connection(s)
I20240503 07:58:22.036134 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:41945
I20240503 07:58:22.037308 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:58:22.047065 1153 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43995
I20240503 07:58:22.047303 1153 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:22.047503 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 1025
I20240503 07:58:22.047600 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:58:22.047942 1153 heartbeater.cc:511] Master 127.0.99.190:43995 requested a full tablet report, sending...
I20240503 07:58:22.048935 441 ts_manager.cc:194] Registered new tserver with Master: 0a385716c25a4f9d99f9aa81cd521d36 (127.0.99.133:41945)
I20240503 07:58:22.049504 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:47899
I20240503 07:58:22.051276 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:58:22.063279 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43818:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 07:58:22.082207 956 tablet_service.cc:1467] Processing CreateTablet for tablet 9d49db6cc76140c8a8fec58029b1430b (DEFAULT_TABLE table=TestTable [id=cb2fa6aee1b1464c89dab712c2be2e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:22.082209 694 tablet_service.cc:1467] Processing CreateTablet for tablet 9d49db6cc76140c8a8fec58029b1430b (DEFAULT_TABLE table=TestTable [id=cb2fa6aee1b1464c89dab712c2be2e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:22.083231 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d49db6cc76140c8a8fec58029b1430b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:22.083365 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d49db6cc76140c8a8fec58029b1430b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:22.083302 563 tablet_service.cc:1467] Processing CreateTablet for tablet 9d49db6cc76140c8a8fec58029b1430b (DEFAULT_TABLE table=TestTable [id=cb2fa6aee1b1464c89dab712c2be2e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:22.084230 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d49db6cc76140c8a8fec58029b1430b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:22.088702 1171 tablet_bootstrap.cc:492] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f: Bootstrap starting.
I20240503 07:58:22.089865 1172 tablet_bootstrap.cc:492] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f: Bootstrap starting.
I20240503 07:58:22.089946 1173 tablet_bootstrap.cc:492] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44: Bootstrap starting.
I20240503 07:58:22.091828 1172 tablet_bootstrap.cc:654] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:22.091905 1171 tablet_bootstrap.cc:654] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:22.091905 1173 tablet_bootstrap.cc:654] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:22.092521 1172 log.cc:824] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:22.092651 1171 log.cc:824] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:22.092651 1173 log.cc:824] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:22.094449 1172 tablet_bootstrap.cc:492] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f: No bootstrap required, opened a new log
I20240503 07:58:22.094456 1173 tablet_bootstrap.cc:492] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44: No bootstrap required, opened a new log
I20240503 07:58:22.094461 1171 tablet_bootstrap.cc:492] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f: No bootstrap required, opened a new log
I20240503 07:58:22.094635 1173 ts_tablet_manager.cc:1367] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:58:22.094635 1172 ts_tablet_manager.cc:1367] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:58:22.094636 1171 ts_tablet_manager.cc:1367] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 07:58:22.098003 1173 raft_consensus.cc:348] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff088811de40ebb8baa70d2db6195f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45277 } } peers { permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 } } peers { permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 } }
I20240503 07:58:22.098003 1172 raft_consensus.cc:348] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff088811de40ebb8baa70d2db6195f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45277 } } peers { permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 } } peers { permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 } }
I20240503 07:58:22.098263 1173 raft_consensus.cc:374] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:22.098263 1172 raft_consensus.cc:374] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:22.098191 1171 raft_consensus.cc:348] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff088811de40ebb8baa70d2db6195f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45277 } } peers { permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 } } peers { permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 } }
I20240503 07:58:22.098323 1173 raft_consensus.cc:729] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ada3d57094474183654dc5e16d5d44, State: Initialized, Role: FOLLOWER
I20240503 07:58:22.098410 1172 raft_consensus.cc:729] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ff088811de40ebb8baa70d2db6195f, State: Initialized, Role: FOLLOWER
I20240503 07:58:22.098412 1171 raft_consensus.cc:374] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:22.098472 1171 raft_consensus.cc:729] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27074610401e48ca898230872e848b7f, State: Initialized, Role: FOLLOWER
I20240503 07:58:22.098843 1171 consensus_queue.cc:260] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff088811de40ebb8baa70d2db6195f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45277 } } peers { permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 } } peers { permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 } }
I20240503 07:58:22.098848 1173 consensus_queue.cc:260] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff088811de40ebb8baa70d2db6195f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45277 } } peers { permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 } } peers { permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 } }
I20240503 07:58:22.098848 1172 consensus_queue.cc:260] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff088811de40ebb8baa70d2db6195f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45277 } } peers { permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 } } peers { permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 } }
I20240503 07:58:22.099659 1171 ts_tablet_manager.cc:1398] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:58:22.099705 1173 ts_tablet_manager.cc:1398] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:58:22.099705 1172 ts_tablet_manager.cc:1398] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:58:22.100047 1022 heartbeater.cc:503] Master 127.0.99.190:43995 was elected leader, sending a full tablet report...
I20240503 07:58:22.100047 760 heartbeater.cc:503] Master 127.0.99.190:43995 was elected leader, sending a full tablet report...
I20240503 07:58:22.100047 629 heartbeater.cc:503] Master 127.0.99.190:43995 was elected leader, sending a full tablet report...
W20240503 07:58:22.189000 761 tablet.cc:2369] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:22.219581 1178 raft_consensus.cc:482] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:22.219799 1178 raft_consensus.cc:504] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff088811de40ebb8baa70d2db6195f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45277 } } peers { permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 } } peers { permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 } }
I20240503 07:58:22.221020 1178 leader_election.cc:288] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ada3d57094474183654dc5e16d5d44 (127.0.99.129:42775), 27074610401e48ca898230872e848b7f (127.0.99.132:44843)
I20240503 07:58:22.221504 1179 raft_consensus.cc:482] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:22.221791 1179 raft_consensus.cc:504] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff088811de40ebb8baa70d2db6195f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45277 } } peers { permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 } } peers { permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 } }
I20240503 07:58:22.223125 1179 leader_election.cc:288] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7ff088811de40ebb8baa70d2db6195f (127.0.99.130:45277), 27074610401e48ca898230872e848b7f (127.0.99.132:44843)
I20240503 07:58:22.226154 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d49db6cc76140c8a8fec58029b1430b" candidate_uuid: "d7ff088811de40ebb8baa70d2db6195f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27074610401e48ca898230872e848b7f" is_pre_election: true
I20240503 07:58:22.226539 976 raft_consensus.cc:2454] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ff088811de40ebb8baa70d2db6195f in term 0.
I20240503 07:58:22.227149 650 leader_election.cc:302] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27074610401e48ca898230872e848b7f, d7ff088811de40ebb8baa70d2db6195f; no voters:
I20240503 07:58:22.227589 1178 raft_consensus.cc:2781] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:22.227671 1178 raft_consensus.cc:482] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:22.227711 1178 raft_consensus.cc:3037] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:22.227849 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d49db6cc76140c8a8fec58029b1430b" candidate_uuid: "d7ff088811de40ebb8baa70d2db6195f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ada3d57094474183654dc5e16d5d44" is_pre_election: true
I20240503 07:58:22.228049 583 raft_consensus.cc:2454] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ff088811de40ebb8baa70d2db6195f in term 0.
I20240503 07:58:22.228825 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d49db6cc76140c8a8fec58029b1430b" candidate_uuid: "99ada3d57094474183654dc5e16d5d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27074610401e48ca898230872e848b7f" is_pre_election: true
I20240503 07:58:22.228968 976 raft_consensus.cc:2454] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ada3d57094474183654dc5e16d5d44 in term 0.
I20240503 07:58:22.229354 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d49db6cc76140c8a8fec58029b1430b" candidate_uuid: "99ada3d57094474183654dc5e16d5d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ff088811de40ebb8baa70d2db6195f" is_pre_election: true
I20240503 07:58:22.229362 519 leader_election.cc:302] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27074610401e48ca898230872e848b7f, 99ada3d57094474183654dc5e16d5d44; no voters:
I20240503 07:58:22.229537 1178 raft_consensus.cc:504] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff088811de40ebb8baa70d2db6195f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45277 } } peers { permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 } } peers { permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 } }
I20240503 07:58:22.229656 1179 raft_consensus.cc:2781] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:22.229745 1179 raft_consensus.cc:482] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:22.229794 1179 raft_consensus.cc:3037] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:22.229799 714 raft_consensus.cc:2379] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99ada3d57094474183654dc5e16d5d44 in current term 1: Already voted for candidate d7ff088811de40ebb8baa70d2db6195f in this term.
I20240503 07:58:22.230296 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d49db6cc76140c8a8fec58029b1430b" candidate_uuid: "d7ff088811de40ebb8baa70d2db6195f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27074610401e48ca898230872e848b7f"
I20240503 07:58:22.230484 976 raft_consensus.cc:3037] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:22.230545 1178 leader_election.cc:288] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [CANDIDATE]: Term 1 election: Requested vote from peers 99ada3d57094474183654dc5e16d5d44 (127.0.99.129:42775), 27074610401e48ca898230872e848b7f (127.0.99.132:44843)
I20240503 07:58:22.230728 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d49db6cc76140c8a8fec58029b1430b" candidate_uuid: "d7ff088811de40ebb8baa70d2db6195f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ada3d57094474183654dc5e16d5d44"
I20240503 07:58:22.230975 1179 raft_consensus.cc:504] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff088811de40ebb8baa70d2db6195f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45277 } } peers { permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 } } peers { permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 } }
I20240503 07:58:22.231218 583 raft_consensus.cc:2379] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d7ff088811de40ebb8baa70d2db6195f in current term 1: Already voted for candidate 99ada3d57094474183654dc5e16d5d44 in this term.
I20240503 07:58:22.231372 1179 leader_election.cc:288] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [CANDIDATE]: Term 1 election: Requested vote from peers d7ff088811de40ebb8baa70d2db6195f (127.0.99.130:45277), 27074610401e48ca898230872e848b7f (127.0.99.132:44843)
I20240503 07:58:22.231552 976 raft_consensus.cc:2454] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ff088811de40ebb8baa70d2db6195f in term 1.
I20240503 07:58:22.231711 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d49db6cc76140c8a8fec58029b1430b" candidate_uuid: "99ada3d57094474183654dc5e16d5d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ff088811de40ebb8baa70d2db6195f"
I20240503 07:58:22.231789 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d49db6cc76140c8a8fec58029b1430b" candidate_uuid: "99ada3d57094474183654dc5e16d5d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27074610401e48ca898230872e848b7f"
I20240503 07:58:22.231842 714 raft_consensus.cc:2379] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99ada3d57094474183654dc5e16d5d44 in current term 1: Already voted for candidate d7ff088811de40ebb8baa70d2db6195f in this term.
I20240503 07:58:22.231858 650 leader_election.cc:302] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27074610401e48ca898230872e848b7f, d7ff088811de40ebb8baa70d2db6195f; no voters:
I20240503 07:58:22.231921 976 raft_consensus.cc:2379] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99ada3d57094474183654dc5e16d5d44 in current term 1: Already voted for candidate d7ff088811de40ebb8baa70d2db6195f in this term.
I20240503 07:58:22.232014 1178 raft_consensus.cc:2781] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:22.232246 1178 raft_consensus.cc:686] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [term 1 LEADER]: Becoming Leader. State: Replica: d7ff088811de40ebb8baa70d2db6195f, State: Running, Role: LEADER
I20240503 07:58:22.232381 516 leader_election.cc:302] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99ada3d57094474183654dc5e16d5d44; no voters: 27074610401e48ca898230872e848b7f, d7ff088811de40ebb8baa70d2db6195f
I20240503 07:58:22.232589 1179 raft_consensus.cc:2726] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 07:58:22.232605 1178 consensus_queue.cc:237] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff088811de40ebb8baa70d2db6195f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45277 } } peers { permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 } } peers { permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 } }
I20240503 07:58:22.235862 440 catalog_manager.cc:5484] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f reported cstate change: term changed from 0 to 1, leader changed from <none> to d7ff088811de40ebb8baa70d2db6195f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d7ff088811de40ebb8baa70d2db6195f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff088811de40ebb8baa70d2db6195f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:22.249936 630 tablet.cc:2369] T 9d49db6cc76140c8a8fec58029b1430b P 99ada3d57094474183654dc5e16d5d44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:22.269145 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:58:22.271015 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 99ada3d57094474183654dc5e16d5d44 to finish bootstrapping
I20240503 07:58:22.276680 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d7ff088811de40ebb8baa70d2db6195f to finish bootstrapping
I20240503 07:58:22.281684 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e6b2fee74eb046268f08c397f8885bf7 to finish bootstrapping
I20240503 07:58:22.287814 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 27074610401e48ca898230872e848b7f to finish bootstrapping
I20240503 07:58:22.292711 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0a385716c25a4f9d99f9aa81cd521d36 to finish bootstrapping
W20240503 07:58:22.320415 1023 tablet.cc:2369] T 9d49db6cc76140c8a8fec58029b1430b P 27074610401e48ca898230872e848b7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:22.497274 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:22.497285 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:22.497257 694 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:22.499603 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:22.499634 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:22.594471 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:22.594878 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:22.594996 694 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:22.595433 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:22.612303 1178 consensus_queue.cc:1035] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [LEADER]: Connected to new peer: Peer: permanent_uuid: "27074610401e48ca898230872e848b7f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:22.619774 1185 consensus_queue.cc:1035] T 9d49db6cc76140c8a8fec58029b1430b P d7ff088811de40ebb8baa70d2db6195f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ada3d57094474183654dc5e16d5d44" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:22.627326 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:22.643998 891 heartbeater.cc:503] Master 127.0.99.190:43995 was elected leader, sending a full tablet report...
I20240503 07:58:22.653182 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 501
I20240503 07:58:22.661231 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 632
I20240503 07:58:22.666915 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 763
I20240503 07:58:22.672621 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 894
I20240503 07:58:22.679381 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 1025
I20240503 07:58:22.684886 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 410
2024-05-03T07:58:22Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (1919 ms)
[----------] 1 test from AdminCliTest (1919 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240503 07:58:22.706753 398 test_util.cc:268] Using random seed: 359451706
I20240503 07:58:22.707546 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:58:22.707592 398 ts_itest-base.cc:113] --------------
I20240503 07:58:22.707602 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 07:58:22.707612 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:58:22.707620 398 ts_itest-base.cc:116] --------------
2024-05-03T07:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:58:22Z Disabled control of system clock
I20240503 07:58:22.715503 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:32841
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:32841
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 07:58:22.828591 1270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:22.828884 1270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:22.828943 1270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:22.832213 1270 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:22.832325 1270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:58:22.832353 1270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:22.832381 1270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:58:22.832418 1270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:58:22.836503 1270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:32841
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:32841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:22.837620 1270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:22.838938 1270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:22.847296 1278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:22.847656 1275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:22.848169 1276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:22.848709 1270 server_base.cc:1034] running on GCE node
I20240503 07:58:22.849299 1270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:22.850399 1270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:22.851588 1270 hybrid_clock.cc:648] HybridClock initialized: now 1714723102851574 us; error 35 us; skew 500 ppm
I20240503 07:58:22.853809 1270 webserver.cc:460] Webserver started at http://127.0.99.190:37413/ using document root <none> and password file <none>
I20240503 07:58:22.854480 1270 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:22.854542 1270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:22.854751 1270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:22.856619 1270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "fc721187c3ca423fae6de6826ff02b7e"
format_stamp: "Formatted at 2024-05-03 07:58:22 on dist-test-slave-bj8b"
I20240503 07:58:22.857137 1270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "fc721187c3ca423fae6de6826ff02b7e"
format_stamp: "Formatted at 2024-05-03 07:58:22 on dist-test-slave-bj8b"
I20240503 07:58:22.860925 1270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 07:58:22.863350 1284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:22.864539 1270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:22.864671 1270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "fc721187c3ca423fae6de6826ff02b7e"
format_stamp: "Formatted at 2024-05-03 07:58:22 on dist-test-slave-bj8b"
I20240503 07:58:22.864769 1270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:22.883378 1270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:22.884122 1270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:22.884296 1270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:22.891086 1270 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:32841
I20240503 07:58:22.891096 1336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:32841 every 8 connection(s)
I20240503 07:58:22.892163 1270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:58:22.895001 1337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:22.900411 1337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e: Bootstrap starting.
I20240503 07:58:22.900970 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 1270
I20240503 07:58:22.901087 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:58:22.903111 1337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:22.903990 1337 log.cc:824] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:22.905815 1337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e: No bootstrap required, opened a new log
I20240503 07:58:22.908727 1337 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc721187c3ca423fae6de6826ff02b7e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 32841 } }
I20240503 07:58:22.908962 1337 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:22.909020 1337 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc721187c3ca423fae6de6826ff02b7e, State: Initialized, Role: FOLLOWER
I20240503 07:58:22.909480 1337 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc721187c3ca423fae6de6826ff02b7e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 32841 } }
I20240503 07:58:22.909608 1337 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:22.909669 1337 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:22.909785 1337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:22.910663 1337 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc721187c3ca423fae6de6826ff02b7e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 32841 } }
I20240503 07:58:22.910992 1337 leader_election.cc:302] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc721187c3ca423fae6de6826ff02b7e; no voters:
I20240503 07:58:22.911273 1337 leader_election.cc:288] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:22.911356 1342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:22.911566 1342 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [term 1 LEADER]: Becoming Leader. State: Replica: fc721187c3ca423fae6de6826ff02b7e, State: Running, Role: LEADER
I20240503 07:58:22.911859 1342 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc721187c3ca423fae6de6826ff02b7e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 32841 } }
I20240503 07:58:22.912443 1337 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:58:22.913553 1344 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc721187c3ca423fae6de6826ff02b7e. Latest consensus state: current_term: 1 leader_uuid: "fc721187c3ca423fae6de6826ff02b7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc721187c3ca423fae6de6826ff02b7e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 32841 } } }
I20240503 07:58:22.913669 1344 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:22.913966 1343 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc721187c3ca423fae6de6826ff02b7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc721187c3ca423fae6de6826ff02b7e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 32841 } } }
I20240503 07:58:22.914062 1343 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e [sys.catalog]: This master's current role is: LEADER
W20240503 07:58:22.918128 1357 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 07:58:22.918483 1357 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 07:58:22.918557 1349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:58:22.919492 1349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:58:22.925104 1349 catalog_manager.cc:1347] Generated new cluster ID: c08093141cd34f74a6a3e3d86b545e37
I20240503 07:58:22.925177 1349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:58:22.936158 1349 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:58:22.937165 1349 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:58:22.957423 1349 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fc721187c3ca423fae6de6826ff02b7e: Generated new TSK 0
I20240503 07:58:22.958232 1349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:58:22.964598 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32841
--builtin_ntp_servers=127.0.99.148:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 07:58:23.080070 1361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:23.080322 1361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:23.080372 1361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:23.083572 1361 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:23.083652 1361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:23.083752 1361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:58:23.087625 1361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32841
--never_fsync=true
--heap_profile_path=/tmp/kudu.1361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:23.088681 1361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:23.089923 1361 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:23.099006 1367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:23.099424 1366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:23.099856 1369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:23.100406 1361 server_base.cc:1034] running on GCE node
I20240503 07:58:23.100845 1361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:23.101460 1361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:23.102625 1361 hybrid_clock.cc:648] HybridClock initialized: now 1714723103102607 us; error 34 us; skew 500 ppm
I20240503 07:58:23.104741 1361 webserver.cc:460] Webserver started at http://127.0.99.129:41451/ using document root <none> and password file <none>
I20240503 07:58:23.105319 1361 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:23.105376 1361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:23.105573 1361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:23.107487 1361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7147310633b54b2ea08416b675a9ebc8"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.107996 1361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7147310633b54b2ea08416b675a9ebc8"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.111649 1361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:23.114106 1375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.115248 1361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:23.115363 1361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7147310633b54b2ea08416b675a9ebc8"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.115463 1361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:23.144414 1361 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:23.145146 1361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:23.145318 1361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:23.145906 1361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:23.146891 1361 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:23.146945 1361 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.147006 1361 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:23.147029 1361 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.155395 1361 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:39851
I20240503 07:58:23.155421 1488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:39851 every 8 connection(s)
I20240503 07:58:23.156412 1361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:58:23.160627 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 1361
I20240503 07:58:23.160724 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:58:23.163085 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32841
--builtin_ntp_servers=127.0.99.148:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:23.166152 1489 heartbeater.cc:347] Connected to a master server at 127.0.99.190:32841
I20240503 07:58:23.166453 1489 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:23.167006 1489 heartbeater.cc:511] Master 127.0.99.190:32841 requested a full tablet report, sending...
I20240503 07:58:23.168614 1301 ts_manager.cc:194] Registered new tserver with Master: 7147310633b54b2ea08416b675a9ebc8 (127.0.99.129:39851)
I20240503 07:58:23.169993 1301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:60187
W20240503 07:58:23.277729 1492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:23.278019 1492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:23.278088 1492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:23.281649 1492 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:23.281756 1492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:23.281854 1492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:58:23.285828 1492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32841
--never_fsync=true
--heap_profile_path=/tmp/kudu.1492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:23.286924 1492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:23.288527 1492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:23.298084 1497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:23.298517 1500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:23.299046 1498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:23.299639 1492 server_base.cc:1034] running on GCE node
I20240503 07:58:23.300072 1492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:23.300717 1492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:23.301901 1492 hybrid_clock.cc:648] HybridClock initialized: now 1714723103301878 us; error 39 us; skew 500 ppm
I20240503 07:58:23.304168 1492 webserver.cc:460] Webserver started at http://127.0.99.130:43849/ using document root <none> and password file <none>
I20240503 07:58:23.304828 1492 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:23.304885 1492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:23.305070 1492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:23.307054 1492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "02d2a741cfed49bcb7303b6d30f9a077"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.307601 1492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "02d2a741cfed49bcb7303b6d30f9a077"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.311492 1492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 07:58:23.314096 1506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.315488 1492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:23.315637 1492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "02d2a741cfed49bcb7303b6d30f9a077"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.315737 1492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:23.336823 1492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:23.337551 1492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:23.337704 1492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:23.338279 1492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:23.339340 1492 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:23.339393 1492 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.339437 1492 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:23.339457 1492 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.347862 1492 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:39123
I20240503 07:58:23.347882 1619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:39123 every 8 connection(s)
I20240503 07:58:23.348912 1492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:58:23.349622 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 1492
I20240503 07:58:23.349722 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:58:23.352114 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32841
--builtin_ntp_servers=127.0.99.148:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:23.358790 1620 heartbeater.cc:347] Connected to a master server at 127.0.99.190:32841
I20240503 07:58:23.359038 1620 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:23.359532 1620 heartbeater.cc:511] Master 127.0.99.190:32841 requested a full tablet report, sending...
I20240503 07:58:23.360631 1301 ts_manager.cc:194] Registered new tserver with Master: 02d2a741cfed49bcb7303b6d30f9a077 (127.0.99.130:39123)
I20240503 07:58:23.361426 1301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:51241
W20240503 07:58:23.469051 1623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:23.469319 1623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:23.469385 1623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:23.472725 1623 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:23.472818 1623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:23.472918 1623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:58:23.476939 1623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32841
--never_fsync=true
--heap_profile_path=/tmp/kudu.1623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:23.478058 1623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:23.479414 1623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:23.488866 1629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:23.489243 1628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:23.489742 1631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:23.490279 1623 server_base.cc:1034] running on GCE node
I20240503 07:58:23.490772 1623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:23.491465 1623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:23.492614 1623 hybrid_clock.cc:648] HybridClock initialized: now 1714723103492597 us; error 34 us; skew 500 ppm
I20240503 07:58:23.494858 1623 webserver.cc:460] Webserver started at http://127.0.99.131:45119/ using document root <none> and password file <none>
I20240503 07:58:23.495477 1623 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:23.495548 1623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:23.495730 1623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:23.497540 1623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e2152d7eac6e4c9f8a980d03f4a9921a"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.498039 1623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e2152d7eac6e4c9f8a980d03f4a9921a"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.501632 1623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 07:58:23.503993 1637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.505121 1623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 07:58:23.505254 1623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e2152d7eac6e4c9f8a980d03f4a9921a"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.505347 1623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:23.534446 1623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:23.535135 1623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:23.535301 1623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:23.535868 1623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:23.536824 1623 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:23.536883 1623 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.536939 1623 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:23.536963 1623 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.545372 1623 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:46731
I20240503 07:58:23.545387 1750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:46731 every 8 connection(s)
I20240503 07:58:23.546414 1623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:58:23.549248 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 1623
I20240503 07:58:23.549351 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:58:23.551795 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32841
--builtin_ntp_servers=127.0.99.148:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:23.556119 1751 heartbeater.cc:347] Connected to a master server at 127.0.99.190:32841
I20240503 07:58:23.556358 1751 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:23.556852 1751 heartbeater.cc:511] Master 127.0.99.190:32841 requested a full tablet report, sending...
I20240503 07:58:23.557893 1301 ts_manager.cc:194] Registered new tserver with Master: e2152d7eac6e4c9f8a980d03f4a9921a (127.0.99.131:46731)
I20240503 07:58:23.558508 1301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:50493
W20240503 07:58:23.666970 1754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:23.667229 1754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:23.667292 1754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:23.670583 1754 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:23.670670 1754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:23.670760 1754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:58:23.674628 1754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32841
--never_fsync=true
--heap_profile_path=/tmp/kudu.1754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:23.675627 1754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:23.676824 1754 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:23.685716 1760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:23.686208 1759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:23.686563 1762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:23.687098 1754 server_base.cc:1034] running on GCE node
I20240503 07:58:23.687501 1754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:23.688117 1754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:23.689280 1754 hybrid_clock.cc:648] HybridClock initialized: now 1714723103689259 us; error 36 us; skew 500 ppm
I20240503 07:58:23.691422 1754 webserver.cc:460] Webserver started at http://127.0.99.132:34231/ using document root <none> and password file <none>
I20240503 07:58:23.692039 1754 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:23.692097 1754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:23.692276 1754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:23.694108 1754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f20baf6b632f4be1bc6cda5cc14e7659"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.694689 1754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f20baf6b632f4be1bc6cda5cc14e7659"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.698379 1754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:23.700868 1768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.702013 1754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:23.702131 1754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f20baf6b632f4be1bc6cda5cc14e7659"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.702224 1754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:23.732156 1754 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:23.732930 1754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:23.733106 1754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:23.733709 1754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:23.734764 1754 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:23.734817 1754 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.734871 1754 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:23.734899 1754 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.743301 1754 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43237
I20240503 07:58:23.743366 1881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43237 every 8 connection(s)
I20240503 07:58:23.744331 1754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:58:23.748252 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 1754
I20240503 07:58:23.748409 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:58:23.750749 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32841
--builtin_ntp_servers=127.0.99.148:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:23.754552 1882 heartbeater.cc:347] Connected to a master server at 127.0.99.190:32841
I20240503 07:58:23.754837 1882 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:23.755378 1882 heartbeater.cc:511] Master 127.0.99.190:32841 requested a full tablet report, sending...
I20240503 07:58:23.756477 1301 ts_manager.cc:194] Registered new tserver with Master: f20baf6b632f4be1bc6cda5cc14e7659 (127.0.99.132:43237)
I20240503 07:58:23.757057 1301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:38315
W20240503 07:58:23.863318 1885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:23.863592 1885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:23.863653 1885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:23.866838 1885 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:23.866919 1885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:23.867007 1885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:58:23.870929 1885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32841
--never_fsync=true
--heap_profile_path=/tmp/kudu.1885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:23.871954 1885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:23.873189 1885 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:23.882287 1891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:23.882752 1890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:23.883287 1893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:23.883832 1885 server_base.cc:1034] running on GCE node
I20240503 07:58:23.884259 1885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:23.884871 1885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:23.886022 1885 hybrid_clock.cc:648] HybridClock initialized: now 1714723103885998 us; error 42 us; skew 500 ppm
I20240503 07:58:23.888238 1885 webserver.cc:460] Webserver started at http://127.0.99.133:36265/ using document root <none> and password file <none>
I20240503 07:58:23.888826 1885 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:23.888881 1885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:23.889070 1885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:23.890894 1885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c08148a49db7445b8936a320e4d76635"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.891456 1885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c08148a49db7445b8936a320e4d76635"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.895663 1885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 07:58:23.898413 1899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.899688 1885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240503 07:58:23.899816 1885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c08148a49db7445b8936a320e4d76635"
format_stamp: "Formatted at 2024-05-03 07:58:23 on dist-test-slave-bj8b"
I20240503 07:58:23.899925 1885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:23.918560 1885 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:23.919378 1885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:23.919561 1885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:23.920120 1885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:23.921074 1885 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:23.921124 1885 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.921180 1885 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:23.921202 1885 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:23.929937 1885 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:45363
I20240503 07:58:23.929965 2012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:45363 every 8 connection(s)
I20240503 07:58:23.930970 1885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:58:23.937148 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 1885
I20240503 07:58:23.937264 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:58:23.940052 2013 heartbeater.cc:347] Connected to a master server at 127.0.99.190:32841
I20240503 07:58:23.940282 2013 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:23.940754 2013 heartbeater.cc:511] Master 127.0.99.190:32841 requested a full tablet report, sending...
I20240503 07:58:23.941633 1301 ts_manager.cc:194] Registered new tserver with Master: c08148a49db7445b8936a320e4d76635 (127.0.99.133:45363)
I20240503 07:58:23.942199 1301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:39645
I20240503 07:58:23.951613 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:58:23.960216 1301 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44576:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 07:58:23.979475 1947 tablet_service.cc:1467] Processing CreateTablet for tablet b1a9bfe122464153a8bf1eaf824edf8e (DEFAULT_TABLE table=TestTable [id=9aeb2053c9ff41568c379b4418083725]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:23.979475 1423 tablet_service.cc:1467] Processing CreateTablet for tablet b1a9bfe122464153a8bf1eaf824edf8e (DEFAULT_TABLE table=TestTable [id=9aeb2053c9ff41568c379b4418083725]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:23.980607 1947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a9bfe122464153a8bf1eaf824edf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:23.980607 1423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a9bfe122464153a8bf1eaf824edf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:23.984077 1554 tablet_service.cc:1467] Processing CreateTablet for tablet b1a9bfe122464153a8bf1eaf824edf8e (DEFAULT_TABLE table=TestTable [id=9aeb2053c9ff41568c379b4418083725]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:23.984994 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a9bfe122464153a8bf1eaf824edf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:23.988404 2031 tablet_bootstrap.cc:492] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8: Bootstrap starting.
I20240503 07:58:23.988422 2032 tablet_bootstrap.cc:492] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635: Bootstrap starting.
I20240503 07:58:23.990325 2032 tablet_bootstrap.cc:654] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:23.990468 2031 tablet_bootstrap.cc:654] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:23.991026 2033 tablet_bootstrap.cc:492] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077: Bootstrap starting.
I20240503 07:58:23.991205 2031 log.cc:824] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:23.991209 2032 log.cc:824] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:23.993113 2031 tablet_bootstrap.cc:492] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8: No bootstrap required, opened a new log
I20240503 07:58:23.993113 2032 tablet_bootstrap.cc:492] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635: No bootstrap required, opened a new log
I20240503 07:58:23.993299 2032 ts_tablet_manager.cc:1367] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 07:58:23.993299 2031 ts_tablet_manager.cc:1367] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 07:58:23.993595 2033 tablet_bootstrap.cc:654] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:23.994331 2033 log.cc:824] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:23.995893 2033 tablet_bootstrap.cc:492] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077: No bootstrap required, opened a new log
I20240503 07:58:23.996058 2033 ts_tablet_manager.cc:1367] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 07:58:23.997184 2032 raft_consensus.cc:348] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d2a741cfed49bcb7303b6d30f9a077" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39123 } } peers { permanent_uuid: "c08148a49db7445b8936a320e4d76635" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45363 } } peers { permanent_uuid: "7147310633b54b2ea08416b675a9ebc8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39851 } }
I20240503 07:58:23.997264 2031 raft_consensus.cc:348] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d2a741cfed49bcb7303b6d30f9a077" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39123 } } peers { permanent_uuid: "c08148a49db7445b8936a320e4d76635" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45363 } } peers { permanent_uuid: "7147310633b54b2ea08416b675a9ebc8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39851 } }
I20240503 07:58:23.997422 2031 raft_consensus.cc:374] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:23.997421 2032 raft_consensus.cc:374] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:23.997474 2031 raft_consensus.cc:729] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7147310633b54b2ea08416b675a9ebc8, State: Initialized, Role: FOLLOWER
I20240503 07:58:23.997483 2032 raft_consensus.cc:729] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08148a49db7445b8936a320e4d76635, State: Initialized, Role: FOLLOWER
I20240503 07:58:23.997938 2031 consensus_queue.cc:260] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d2a741cfed49bcb7303b6d30f9a077" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39123 } } peers { permanent_uuid: "c08148a49db7445b8936a320e4d76635" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45363 } } peers { permanent_uuid: "7147310633b54b2ea08416b675a9ebc8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39851 } }
I20240503 07:58:23.997938 2032 consensus_queue.cc:260] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d2a741cfed49bcb7303b6d30f9a077" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39123 } } peers { permanent_uuid: "c08148a49db7445b8936a320e4d76635" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45363 } } peers { permanent_uuid: "7147310633b54b2ea08416b675a9ebc8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39851 } }
I20240503 07:58:23.998862 2031 ts_tablet_manager.cc:1398] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240503 07:58:23.998862 2032 ts_tablet_manager.cc:1398] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240503 07:58:23.999136 2033 raft_consensus.cc:348] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d2a741cfed49bcb7303b6d30f9a077" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39123 } } peers { permanent_uuid: "c08148a49db7445b8936a320e4d76635" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45363 } } peers { permanent_uuid: "7147310633b54b2ea08416b675a9ebc8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39851 } }
I20240503 07:58:23.999372 2033 raft_consensus.cc:374] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:23.999373 2013 heartbeater.cc:503] Master 127.0.99.190:32841 was elected leader, sending a full tablet report...
I20240503 07:58:23.999442 2033 raft_consensus.cc:729] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d2a741cfed49bcb7303b6d30f9a077, State: Initialized, Role: FOLLOWER
I20240503 07:58:23.999898 2033 consensus_queue.cc:260] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d2a741cfed49bcb7303b6d30f9a077" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39123 } } peers { permanent_uuid: "c08148a49db7445b8936a320e4d76635" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45363 } } peers { permanent_uuid: "7147310633b54b2ea08416b675a9ebc8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39851 } }
I20240503 07:58:24.000634 1489 heartbeater.cc:503] Master 127.0.99.190:32841 was elected leader, sending a full tablet report...
I20240503 07:58:24.000763 2033 ts_tablet_manager.cc:1398] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240503 07:58:24.001092 1620 heartbeater.cc:503] Master 127.0.99.190:32841 was elected leader, sending a full tablet report...
W20240503 07:58:24.100342 1621 tablet.cc:2369] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:58:24.157930 1490 tablet.cc:2369] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:24.167551 2039 raft_consensus.cc:482] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:24.167755 2039 raft_consensus.cc:504] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d2a741cfed49bcb7303b6d30f9a077" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39123 } } peers { permanent_uuid: "c08148a49db7445b8936a320e4d76635" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45363 } } peers { permanent_uuid: "7147310633b54b2ea08416b675a9ebc8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39851 } }
I20240503 07:58:24.168946 2039 leader_election.cc:288] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c08148a49db7445b8936a320e4d76635 (127.0.99.133:45363), 7147310633b54b2ea08416b675a9ebc8 (127.0.99.129:39851)
I20240503 07:58:24.172842 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a9bfe122464153a8bf1eaf824edf8e" candidate_uuid: "02d2a741cfed49bcb7303b6d30f9a077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7147310633b54b2ea08416b675a9ebc8" is_pre_election: true
I20240503 07:58:24.172892 1967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a9bfe122464153a8bf1eaf824edf8e" candidate_uuid: "02d2a741cfed49bcb7303b6d30f9a077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c08148a49db7445b8936a320e4d76635" is_pre_election: true
I20240503 07:58:24.173184 1967 raft_consensus.cc:2454] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d2a741cfed49bcb7303b6d30f9a077 in term 0.
I20240503 07:58:24.173183 1443 raft_consensus.cc:2454] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d2a741cfed49bcb7303b6d30f9a077 in term 0.
I20240503 07:58:24.173696 1510 leader_election.cc:302] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d2a741cfed49bcb7303b6d30f9a077, c08148a49db7445b8936a320e4d76635; no voters:
I20240503 07:58:24.173962 2039 raft_consensus.cc:2781] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:24.174036 2039 raft_consensus.cc:482] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:24.174065 2039 raft_consensus.cc:3037] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:24.175061 2039 raft_consensus.cc:504] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d2a741cfed49bcb7303b6d30f9a077" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39123 } } peers { permanent_uuid: "c08148a49db7445b8936a320e4d76635" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45363 } } peers { permanent_uuid: "7147310633b54b2ea08416b675a9ebc8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39851 } }
I20240503 07:58:24.175438 2039 leader_election.cc:288] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [CANDIDATE]: Term 1 election: Requested vote from peers c08148a49db7445b8936a320e4d76635 (127.0.99.133:45363), 7147310633b54b2ea08416b675a9ebc8 (127.0.99.129:39851)
I20240503 07:58:24.175838 1967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a9bfe122464153a8bf1eaf824edf8e" candidate_uuid: "02d2a741cfed49bcb7303b6d30f9a077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c08148a49db7445b8936a320e4d76635"
I20240503 07:58:24.175953 1967 raft_consensus.cc:3037] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:24.176810 1967 raft_consensus.cc:2454] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d2a741cfed49bcb7303b6d30f9a077 in term 1.
I20240503 07:58:24.177078 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a9bfe122464153a8bf1eaf824edf8e" candidate_uuid: "02d2a741cfed49bcb7303b6d30f9a077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7147310633b54b2ea08416b675a9ebc8"
I20240503 07:58:24.177112 1510 leader_election.cc:302] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d2a741cfed49bcb7303b6d30f9a077, c08148a49db7445b8936a320e4d76635; no voters:
I20240503 07:58:24.177191 1443 raft_consensus.cc:3037] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:24.177260 2039 raft_consensus.cc:2781] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:24.177470 2039 raft_consensus.cc:686] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [term 1 LEADER]: Becoming Leader. State: Replica: 02d2a741cfed49bcb7303b6d30f9a077, State: Running, Role: LEADER
I20240503 07:58:24.177814 2039 consensus_queue.cc:237] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d2a741cfed49bcb7303b6d30f9a077" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39123 } } peers { permanent_uuid: "c08148a49db7445b8936a320e4d76635" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45363 } } peers { permanent_uuid: "7147310633b54b2ea08416b675a9ebc8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39851 } }
I20240503 07:58:24.178134 1443 raft_consensus.cc:2454] T b1a9bfe122464153a8bf1eaf824edf8e P 7147310633b54b2ea08416b675a9ebc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d2a741cfed49bcb7303b6d30f9a077 in term 1.
I20240503 07:58:24.180876 1300 catalog_manager.cc:5484] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d2a741cfed49bcb7303b6d30f9a077 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02d2a741cfed49bcb7303b6d30f9a077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d2a741cfed49bcb7303b6d30f9a077" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39123 } } peers { permanent_uuid: "c08148a49db7445b8936a320e4d76635" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45363 } } peers { permanent_uuid: "7147310633b54b2ea08416b675a9ebc8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39851 } } }
W20240503 07:58:24.181838 2014 tablet.cc:2369] T b1a9bfe122464153a8bf1eaf824edf8e P c08148a49db7445b8936a320e4d76635: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:24.208534 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:58:24.210031 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7147310633b54b2ea08416b675a9ebc8 to finish bootstrapping
I20240503 07:58:24.215620 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 02d2a741cfed49bcb7303b6d30f9a077 to finish bootstrapping
I20240503 07:58:24.220497 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e2152d7eac6e4c9f8a980d03f4a9921a to finish bootstrapping
I20240503 07:58:24.225993 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f20baf6b632f4be1bc6cda5cc14e7659 to finish bootstrapping
I20240503 07:58:24.231705 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c08148a49db7445b8936a320e4d76635 to finish bootstrapping
I20240503 07:58:24.236445 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 1361
I20240503 07:58:24.435318 1554 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:24.436249 1947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.436296 1685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.438143 1816 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.512549 1554 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:24.512877 1685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.514122 1947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.514194 1816 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.560534 1751 heartbeater.cc:503] Master 127.0.99.190:32841 was elected leader, sending a full tablet report...
I20240503 07:58:24.613790 1947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.614087 1554 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:24.615321 1685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.615737 1816 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.632143 2039 consensus_queue.cc:1035] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c08148a49db7445b8936a320e4d76635" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 45363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:58:24.637414 1507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39851: connect: Connection refused (error 111)
W20240503 07:58:24.638789 1507 consensus_peers.cc:485] T b1a9bfe122464153a8bf1eaf824edf8e P 02d2a741cfed49bcb7303b6d30f9a077 -> Peer 7147310633b54b2ea08416b675a9ebc8 (127.0.99.129:39851): Couldn't send request to peer 7147310633b54b2ea08416b675a9ebc8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:24.759403 1882 heartbeater.cc:503] Master 127.0.99.190:32841 was elected leader, sending a full tablet report...
I20240503 07:58:24.869683 1554 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:24.870381 1947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.871429 1685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.872156 1816 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.961197 1554 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:24.962149 1685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.962249 1816 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:24.962589 1947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:25.062405 1685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:25.062407 1816 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:25.062855 1554 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:25.063711 1947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:25.098166 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 1492
I20240503 07:58:25.104144 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 1623
I20240503 07:58:25.109397 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 1754
I20240503 07:58:25.115111 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 1885
I20240503 07:58:25.120726 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 1270
2024-05-03T07:58:25Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (2437 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2437 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240503 07:58:25.143956 398 test_util.cc:268] Using random seed: 361888909
I20240503 07:58:25.144786 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:58:25.144834 398 ts_itest-base.cc:113] --------------
I20240503 07:58:25.144846 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 07:58:25.144855 398 ts_itest-base.cc:115] 1 replicas per TS
I20240503 07:58:25.144866 398 ts_itest-base.cc:116] --------------
2024-05-03T07:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:58:25Z Disabled control of system clock
I20240503 07:58:25.154141 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34437
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:34437
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 07:58:25.284850 2233 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 07:58:25.285156 2233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:25.285181 2233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:25.285221 2233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:25.288465 2233 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:25.288563 2233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:58:25.288587 2233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:25.288611 2233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:58:25.288631 2233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:58:25.292714 2233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:34437
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:25.293754 2233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:25.295035 2233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:25.303457 2238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:25.303911 2241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:25.304420 2239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:25.304834 2233 server_base.cc:1034] running on GCE node
I20240503 07:58:25.305488 2233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:25.306710 2233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:25.307965 2233 hybrid_clock.cc:648] HybridClock initialized: now 1714723105307949 us; error 34 us; skew 500 ppm
I20240503 07:58:25.310302 2233 webserver.cc:460] Webserver started at http://127.0.99.190:34813/ using document root <none> and password file <none>
I20240503 07:58:25.310990 2233 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:25.311057 2233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:25.311270 2233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:25.313374 2233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6b1bd3a95fbd45e58255ec3143f75686"
format_stamp: "Formatted at 2024-05-03 07:58:25 on dist-test-slave-bj8b"
I20240503 07:58:25.313949 2233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6b1bd3a95fbd45e58255ec3143f75686"
format_stamp: "Formatted at 2024-05-03 07:58:25 on dist-test-slave-bj8b"
I20240503 07:58:25.317953 2233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20240503 07:58:25.320645 2247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:25.322032 2233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:25.322180 2233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6b1bd3a95fbd45e58255ec3143f75686"
format_stamp: "Formatted at 2024-05-03 07:58:25 on dist-test-slave-bj8b"
I20240503 07:58:25.322293 2233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:25.335713 2233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:25.336480 2233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:25.336658 2233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:25.343813 2233 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:34437
I20240503 07:58:25.344043 2299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:34437 every 8 connection(s)
I20240503 07:58:25.345032 2233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:58:25.348883 2300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:25.350594 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 2233
I20240503 07:58:25.350714 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:58:25.355991 2300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686: Bootstrap starting.
I20240503 07:58:25.358510 2300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:25.359449 2300 log.cc:824] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:25.361554 2300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686: No bootstrap required, opened a new log
I20240503 07:58:25.364291 2300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1bd3a95fbd45e58255ec3143f75686" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34437 } }
I20240503 07:58:25.364516 2300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:25.364562 2300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1bd3a95fbd45e58255ec3143f75686, State: Initialized, Role: FOLLOWER
I20240503 07:58:25.364953 2300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1bd3a95fbd45e58255ec3143f75686" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34437 } }
I20240503 07:58:25.365059 2300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:25.365104 2300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:25.365201 2300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:25.438720 2300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1bd3a95fbd45e58255ec3143f75686" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34437 } }
I20240503 07:58:25.439312 2300 leader_election.cc:302] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b1bd3a95fbd45e58255ec3143f75686; no voters:
I20240503 07:58:25.439692 2300 leader_election.cc:288] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:25.440090 2305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:25.440311 2305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [term 1 LEADER]: Becoming Leader. State: Replica: 6b1bd3a95fbd45e58255ec3143f75686, State: Running, Role: LEADER
I20240503 07:58:25.440645 2305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1bd3a95fbd45e58255ec3143f75686" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34437 } }
I20240503 07:58:25.441037 2300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:58:25.442301 2307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b1bd3a95fbd45e58255ec3143f75686. Latest consensus state: current_term: 1 leader_uuid: "6b1bd3a95fbd45e58255ec3143f75686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1bd3a95fbd45e58255ec3143f75686" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34437 } } }
I20240503 07:58:25.442466 2307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:25.442730 2306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b1bd3a95fbd45e58255ec3143f75686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1bd3a95fbd45e58255ec3143f75686" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34437 } } }
I20240503 07:58:25.442833 2306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686 [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:25.443176 2314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:58:25.446102 2314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:58:25.451707 2314 catalog_manager.cc:1347] Generated new cluster ID: 1f04104ca60748a5b8dd7aff1a6783b0
I20240503 07:58:25.451809 2314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:58:25.473008 2314 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:58:25.474411 2314 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:58:25.488600 2314 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6b1bd3a95fbd45e58255ec3143f75686: Generated new TSK 0
I20240503 07:58:25.489794 2314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:58:25.507375 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 07:58:25.629884 2324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:25.630146 2324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:25.630192 2324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:25.633620 2324 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:25.633705 2324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:25.633793 2324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:58:25.637888 2324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.2324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:25.639021 2324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:25.640360 2324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:25.650043 2330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:25.650498 2332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:25.650926 2329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:25.651518 2324 server_base.cc:1034] running on GCE node
I20240503 07:58:25.651973 2324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:25.652629 2324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:25.653797 2324 hybrid_clock.cc:648] HybridClock initialized: now 1714723105653780 us; error 35 us; skew 500 ppm
I20240503 07:58:25.656116 2324 webserver.cc:460] Webserver started at http://127.0.99.129:42641/ using document root <none> and password file <none>
I20240503 07:58:25.656822 2324 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:25.656881 2324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:25.657089 2324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:25.659188 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9405f157903141179a86c810396a416b"
format_stamp: "Formatted at 2024-05-03 07:58:25 on dist-test-slave-bj8b"
I20240503 07:58:25.659730 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9405f157903141179a86c810396a416b"
format_stamp: "Formatted at 2024-05-03 07:58:25 on dist-test-slave-bj8b"
I20240503 07:58:25.663806 2324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 07:58:25.666831 2338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:25.668148 2324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:25.668278 2324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9405f157903141179a86c810396a416b"
format_stamp: "Formatted at 2024-05-03 07:58:25 on dist-test-slave-bj8b"
I20240503 07:58:25.668385 2324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:25.699712 2324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:25.700567 2324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:25.700762 2324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:25.701428 2324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:25.702487 2324 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:25.702543 2324 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:25.702600 2324 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:25.702626 2324 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:25.711930 2324 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:35381
I20240503 07:58:25.711966 2451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:35381 every 8 connection(s)
I20240503 07:58:25.713023 2324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:58:25.714731 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 2324
I20240503 07:58:25.714856 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:58:25.717133 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:25.723899 2452 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:25.724172 2452 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:25.724749 2452 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:25.726527 2264 ts_manager.cc:194] Registered new tserver with Master: 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:25.727921 2264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:48803
W20240503 07:58:25.836071 2455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:25.836365 2455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:25.836416 2455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:25.840235 2455 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:25.840334 2455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:25.840420 2455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:58:25.845435 2455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.2455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:25.846699 2455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:25.848169 2455 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:25.857776 2461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:25.858158 2460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:25.858705 2463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:25.859275 2455 server_base.cc:1034] running on GCE node
I20240503 07:58:25.859757 2455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:25.860567 2455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:25.861755 2455 hybrid_clock.cc:648] HybridClock initialized: now 1714723105861726 us; error 48 us; skew 500 ppm
I20240503 07:58:25.864053 2455 webserver.cc:460] Webserver started at http://127.0.99.130:39597/ using document root <none> and password file <none>
I20240503 07:58:25.864691 2455 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:25.864754 2455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:25.864964 2455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:25.866933 2455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "80435ecadfcf4ede849c7a97abea952d"
format_stamp: "Formatted at 2024-05-03 07:58:25 on dist-test-slave-bj8b"
I20240503 07:58:25.867457 2455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "80435ecadfcf4ede849c7a97abea952d"
format_stamp: "Formatted at 2024-05-03 07:58:25 on dist-test-slave-bj8b"
I20240503 07:58:25.871304 2455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 07:58:25.873999 2469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:25.875321 2455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 07:58:25.875465 2455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "80435ecadfcf4ede849c7a97abea952d"
format_stamp: "Formatted at 2024-05-03 07:58:25 on dist-test-slave-bj8b"
I20240503 07:58:25.875571 2455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:25.891613 2455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:25.892393 2455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:25.892577 2455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:25.893208 2455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:25.894304 2455 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:25.894376 2455 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:25.894435 2455 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:25.894459 2455 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:25.903481 2455 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:33895
I20240503 07:58:25.903508 2582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:33895 every 8 connection(s)
I20240503 07:58:25.904606 2455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:58:25.914252 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 2455
I20240503 07:58:25.914431 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:58:25.914708 2583 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:25.914978 2583 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:25.915639 2583 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:25.916805 2264 ts_manager.cc:194] Registered new tserver with Master: 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895)
I20240503 07:58:25.916841 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:25.917428 2264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:52877
W20240503 07:58:26.038730 2586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:26.038995 2586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:26.039057 2586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:26.042469 2586 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:26.042551 2586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:26.042646 2586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:58:26.046738 2586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.2586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:26.047911 2586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:26.049201 2586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:26.059010 2591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:26.059460 2592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:26.060061 2594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:26.060628 2586 server_base.cc:1034] running on GCE node
I20240503 07:58:26.061091 2586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:26.061764 2586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:26.062924 2586 hybrid_clock.cc:648] HybridClock initialized: now 1714723106062900 us; error 40 us; skew 500 ppm
I20240503 07:58:26.065028 2586 webserver.cc:460] Webserver started at http://127.0.99.131:33621/ using document root <none> and password file <none>
I20240503 07:58:26.065645 2586 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:26.065702 2586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:26.065904 2586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:26.067778 2586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e4a21231285448c88ef7465d265f20b5"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.068295 2586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e4a21231285448c88ef7465d265f20b5"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.072113 2586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240503 07:58:26.074715 2600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.075965 2586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:26.076094 2586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e4a21231285448c88ef7465d265f20b5"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.076205 2586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:26.100299 2586 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:26.101028 2586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:26.101194 2586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:26.101790 2586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:26.102835 2586 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:26.102895 2586 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.102952 2586 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:26.102977 2586 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.111445 2586 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:38167
I20240503 07:58:26.111460 2713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:38167 every 8 connection(s)
I20240503 07:58:26.112491 2586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:58:26.113242 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 2586
I20240503 07:58:26.113354 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:58:26.115809 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:26.123147 2714 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:26.123453 2714 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:26.124104 2714 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:26.125332 2264 ts_manager.cc:194] Registered new tserver with Master: e4a21231285448c88ef7465d265f20b5 (127.0.99.131:38167)
I20240503 07:58:26.126015 2264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:35395
W20240503 07:58:26.237287 2717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:26.237536 2717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:26.237582 2717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:26.240947 2717 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:26.241036 2717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:26.241129 2717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:58:26.245218 2717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.2717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:26.246285 2717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:26.247611 2717 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:26.257022 2725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:26.257383 2722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:26.257925 2723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:26.258508 2717 server_base.cc:1034] running on GCE node
I20240503 07:58:26.258937 2717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:26.259579 2717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:26.260759 2717 hybrid_clock.cc:648] HybridClock initialized: now 1714723106260729 us; error 39 us; skew 500 ppm
I20240503 07:58:26.262941 2717 webserver.cc:460] Webserver started at http://127.0.99.132:45215/ using document root <none> and password file <none>
I20240503 07:58:26.263566 2717 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:26.263629 2717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:26.263823 2717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:26.265724 2717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1147cda1821744448cfea2164c8f3d58"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.266268 2717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1147cda1821744448cfea2164c8f3d58"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.270067 2717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 07:58:26.272908 2731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.274464 2717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:26.274595 2717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1147cda1821744448cfea2164c8f3d58"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.274704 2717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:26.288033 2717 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:26.288795 2717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:26.288976 2717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:26.289598 2717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:26.290642 2717 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:26.290695 2717 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.290742 2717 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:26.290771 2717 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.299283 2717 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:46551
I20240503 07:58:26.299311 2844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:46551 every 8 connection(s)
I20240503 07:58:26.300290 2717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:58:26.302453 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 2717
I20240503 07:58:26.302552 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:58:26.304602 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:26.310411 2845 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:26.310669 2845 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:26.311196 2845 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:26.312232 2264 ts_manager.cc:194] Registered new tserver with Master: 1147cda1821744448cfea2164c8f3d58 (127.0.99.132:46551)
I20240503 07:58:26.312826 2264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:39669
W20240503 07:58:26.428287 2848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:26.428570 2848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:26.428627 2848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:26.432001 2848 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:26.432077 2848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:26.432166 2848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:58:26.436376 2848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.2848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:26.437430 2848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:26.438710 2848 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:26.447810 2854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:26.448422 2853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:26.448869 2856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:26.449359 2848 server_base.cc:1034] running on GCE node
I20240503 07:58:26.449823 2848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:26.450513 2848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:26.451668 2848 hybrid_clock.cc:648] HybridClock initialized: now 1714723106451637 us; error 54 us; skew 500 ppm
I20240503 07:58:26.453948 2848 webserver.cc:460] Webserver started at http://127.0.99.133:40781/ using document root <none> and password file <none>
I20240503 07:58:26.454730 2848 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:26.454792 2848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:26.454993 2848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:26.456845 2848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6fa4f16dd2a545dbb01a0b260ce68744"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.457360 2848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6fa4f16dd2a545dbb01a0b260ce68744"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.461197 2848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 07:58:26.463577 2862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.464705 2848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:26.464813 2848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6fa4f16dd2a545dbb01a0b260ce68744"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.464905 2848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:26.479480 2848 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:26.480260 2848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:26.480443 2848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:26.481168 2848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:26.482187 2848 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:26.482241 2848 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.482299 2848 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:26.482326 2848 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.491429 2848 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:43115
I20240503 07:58:26.491467 2975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:43115 every 8 connection(s)
I20240503 07:58:26.492590 2848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:58:26.501677 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 2848
I20240503 07:58:26.501814 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:58:26.502255 2976 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:26.502545 2976 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:26.503082 2976 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:26.504050 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:26.504079 2264 ts_manager.cc:194] Registered new tserver with Master: 6fa4f16dd2a545dbb01a0b260ce68744 (127.0.99.133:43115)
I20240503 07:58:26.504665 2264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:57025
W20240503 07:58:26.617178 2979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:26.617460 2979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:26.617509 2979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:26.620754 2979 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:26.620829 2979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:26.620908 2979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:58:26.624835 2979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.2979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:26.625859 2979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:26.627213 2979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:26.636222 2984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:26.636710 2985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:26.637169 2987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:26.637730 2979 server_base.cc:1034] running on GCE node
I20240503 07:58:26.638124 2979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:26.638743 2979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:26.639902 2979 hybrid_clock.cc:648] HybridClock initialized: now 1714723106639882 us; error 33 us; skew 500 ppm
I20240503 07:58:26.641911 2979 webserver.cc:460] Webserver started at http://127.0.99.134:36101/ using document root <none> and password file <none>
I20240503 07:58:26.642556 2979 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:26.642617 2979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:26.642817 2979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:26.644655 2979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6245203dd9bb46a18e8f3b4526d16b82"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.645186 2979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6245203dd9bb46a18e8f3b4526d16b82"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.648746 2979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:58:26.653296 2993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.654613 2979 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20240503 07:58:26.654728 2979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6245203dd9bb46a18e8f3b4526d16b82"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.654834 2979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:26.681782 2979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:26.682572 2979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:26.682737 2979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:26.683326 2979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:26.684324 2979 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:26.684374 2979 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.684419 2979 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:26.684449 2979 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.693006 2979 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:46183
I20240503 07:58:26.693018 3106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:46183 every 8 connection(s)
I20240503 07:58:26.694067 2979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:58:26.700768 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 2979
I20240503 07:58:26.700870 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 07:58:26.703264 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:26.703799 3107 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:26.704034 3107 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:26.704545 3107 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:26.705515 2264 ts_manager.cc:194] Registered new tserver with Master: 6245203dd9bb46a18e8f3b4526d16b82 (127.0.99.134:46183)
I20240503 07:58:26.706122 2264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:43281
I20240503 07:58:26.730645 2452 heartbeater.cc:503] Master 127.0.99.190:34437 was elected leader, sending a full tablet report...
W20240503 07:58:26.818584 3110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:26.818830 3110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:26.818881 3110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:26.822080 3110 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:26.822165 3110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:26.822249 3110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:58:26.826210 3110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.3110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:26.827324 3110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:26.828608 3110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:26.837754 3116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:26.838196 3115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:26.838712 3118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:26.839322 3110 server_base.cc:1034] running on GCE node
I20240503 07:58:26.839742 3110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:26.840364 3110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:26.841533 3110 hybrid_clock.cc:648] HybridClock initialized: now 1714723106841506 us; error 43 us; skew 500 ppm
I20240503 07:58:26.844604 3110 webserver.cc:460] Webserver started at http://127.0.99.135:37461/ using document root <none> and password file <none>
I20240503 07:58:26.845441 3110 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:26.845525 3110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:26.845738 3110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:26.848228 3110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "27715b19d4ad40d89c2d12e66aca6a34"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.848961 3110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "27715b19d4ad40d89c2d12e66aca6a34"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.852845 3110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 07:58:26.856264 3124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.857537 3110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:26.857652 3110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "27715b19d4ad40d89c2d12e66aca6a34"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:26.857762 3110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:26.879889 3110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:26.880736 3110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:26.880916 3110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:26.881513 3110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:26.882573 3110 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:26.882625 3110 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.882669 3110 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:26.882700 3110 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:26.892786 3110 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:34969
I20240503 07:58:26.892830 3237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:34969 every 8 connection(s)
I20240503 07:58:26.893862 3110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:58:26.900065 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 3110
I20240503 07:58:26.900185 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 07:58:26.904186 3238 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:26.904471 3238 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:26.905062 3238 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:26.906296 2264 ts_manager.cc:194] Registered new tserver with Master: 27715b19d4ad40d89c2d12e66aca6a34 (127.0.99.135:34969)
I20240503 07:58:26.906919 2264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:42055
I20240503 07:58:26.914994 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 07:58:26.919606 2583 heartbeater.cc:503] Master 127.0.99.190:34437 was elected leader, sending a full tablet report...
I20240503 07:58:26.924376 2264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49024:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 07:58:26.943646 2648 tablet_service.cc:1467] Processing CreateTablet for tablet 0189e0b9411f41908cbce67fce3ea308 (DEFAULT_TABLE table=TestTable [id=bd15ae1b0ca947a9ab28dba6fbe21c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:26.944805 2648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0189e0b9411f41908cbce67fce3ea308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:26.951958 3252 tablet_bootstrap.cc:492] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5: Bootstrap starting.
I20240503 07:58:26.954319 3252 tablet_bootstrap.cc:654] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:26.955103 3252 log.cc:824] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:26.956974 3252 tablet_bootstrap.cc:492] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5: No bootstrap required, opened a new log
I20240503 07:58:26.957165 3252 ts_tablet_manager.cc:1367] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 07:58:26.960144 3252 raft_consensus.cc:348] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:26.960371 3252 raft_consensus.cc:374] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:26.960439 3252 raft_consensus.cc:729] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a21231285448c88ef7465d265f20b5, State: Initialized, Role: FOLLOWER
I20240503 07:58:26.960880 3252 consensus_queue.cc:260] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:26.961009 3252 raft_consensus.cc:388] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:26.961073 3252 raft_consensus.cc:482] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:26.961185 3252 raft_consensus.cc:3037] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:27.048799 3252 raft_consensus.cc:504] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:27.049384 3252 leader_election.cc:302] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4a21231285448c88ef7465d265f20b5; no voters:
I20240503 07:58:27.049803 3252 leader_election.cc:288] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:27.049945 3254 raft_consensus.cc:2781] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:27.050190 3254 raft_consensus.cc:686] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 1 LEADER]: Becoming Leader. State: Replica: e4a21231285448c88ef7465d265f20b5, State: Running, Role: LEADER
I20240503 07:58:27.050612 2714 heartbeater.cc:503] Master 127.0.99.190:34437 was elected leader, sending a full tablet report...
I20240503 07:58:27.050781 3254 consensus_queue.cc:237] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:27.050891 3252 ts_tablet_manager.cc:1398] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5: Time spent starting tablet: real 0.094s user 0.006s sys 0.002s
I20240503 07:58:27.055546 2264 catalog_manager.cc:5484] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4a21231285448c88ef7465d265f20b5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "e4a21231285448c88ef7465d265f20b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } } }
I20240503 07:58:27.062565 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 07:58:27.064286 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9405f157903141179a86c810396a416b to finish bootstrapping
I20240503 07:58:27.070874 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 80435ecadfcf4ede849c7a97abea952d to finish bootstrapping
I20240503 07:58:27.077136 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e4a21231285448c88ef7465d265f20b5 to finish bootstrapping
I20240503 07:58:27.082605 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1147cda1821744448cfea2164c8f3d58 to finish bootstrapping
I20240503 07:58:27.088522 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6fa4f16dd2a545dbb01a0b260ce68744 to finish bootstrapping
I20240503 07:58:27.094627 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6245203dd9bb46a18e8f3b4526d16b82 to finish bootstrapping
I20240503 07:58:27.100617 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 27715b19d4ad40d89c2d12e66aca6a34 to finish bootstrapping
I20240503 07:58:27.106667 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 2586
I20240503 07:58:27.112267 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 2717
I20240503 07:58:27.117112 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 2848
I20240503 07:58:27.123548 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 2979
I20240503 07:58:27.129659 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 3110
I20240503 07:58:30.886567 2263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49024:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:30.900061 2517 tablet_service.cc:1467] Processing CreateTablet for tablet b46fc192e7aa439d988ee7c4e51603c2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5311ba7c92bf4c08b8ac5d9435e4950f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:30.900063 2385 tablet_service.cc:1467] Processing CreateTablet for tablet 1246a3d482304f02bbfe1c97c2e5a1ad (DEFAULT_TABLE table=rebalance_test_table_0 [id=5311ba7c92bf4c08b8ac5d9435e4950f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:30.900061 2386 tablet_service.cc:1467] Processing CreateTablet for tablet d81db8a9c3da42e7a26ef1c665b6c93b (DEFAULT_TABLE table=rebalance_test_table_0 [id=5311ba7c92bf4c08b8ac5d9435e4950f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:30.901333 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b46fc192e7aa439d988ee7c4e51603c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:30.901332 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81db8a9c3da42e7a26ef1c665b6c93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:30.901774 2385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1246a3d482304f02bbfe1c97c2e5a1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:30.909024 3275 tablet_bootstrap.cc:492] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d: Bootstrap starting.
I20240503 07:58:30.909013 3274 tablet_bootstrap.cc:492] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b: Bootstrap starting.
I20240503 07:58:30.911525 3275 tablet_bootstrap.cc:654] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:30.911525 3274 tablet_bootstrap.cc:654] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:30.912461 3275 log.cc:824] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:30.912461 3274 log.cc:824] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:30.914391 3274 tablet_bootstrap.cc:492] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b: No bootstrap required, opened a new log
I20240503 07:58:30.914492 3275 tablet_bootstrap.cc:492] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d: No bootstrap required, opened a new log
I20240503 07:58:30.914587 3274 ts_tablet_manager.cc:1367] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 07:58:30.914667 3275 ts_tablet_manager.cc:1367] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 07:58:30.917694 3275 raft_consensus.cc:348] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:30.917671 3274 raft_consensus.cc:348] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:30.917914 3275 raft_consensus.cc:374] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:30.917914 3274 raft_consensus.cc:374] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:30.917977 3274 raft_consensus.cc:729] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Initialized, Role: FOLLOWER
I20240503 07:58:30.917997 3275 raft_consensus.cc:729] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Initialized, Role: FOLLOWER
I20240503 07:58:30.918474 3274 consensus_queue.cc:260] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:30.918474 3275 consensus_queue.cc:260] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:30.918605 3275 raft_consensus.cc:388] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:30.918606 3274 raft_consensus.cc:388] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:30.918671 3274 raft_consensus.cc:482] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:30.918670 3275 raft_consensus.cc:482] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:30.918771 3274 raft_consensus.cc:3037] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:30.918772 3275 raft_consensus.cc:3037] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:30.919582 3275 raft_consensus.cc:504] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:30.919595 3274 raft_consensus.cc:504] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:30.919906 3275 leader_election.cc:302] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80435ecadfcf4ede849c7a97abea952d; no voters:
I20240503 07:58:30.919905 3274 leader_election.cc:302] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:30.920197 3274 leader_election.cc:288] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:30.920198 3275 leader_election.cc:288] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:30.920295 3279 raft_consensus.cc:2781] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:30.920305 3278 raft_consensus.cc:2781] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:30.920502 3278 raft_consensus.cc:686] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Leader. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:30.920696 3279 raft_consensus.cc:686] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Becoming Leader. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Running, Role: LEADER
I20240503 07:58:30.920861 3278 consensus_queue.cc:237] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:30.921053 3279 consensus_queue.cc:237] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:30.921123 3274 ts_tablet_manager.cc:1398] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 07:58:30.921123 3275 ts_tablet_manager.cc:1398] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 07:58:30.921507 3274 tablet_bootstrap.cc:492] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b: Bootstrap starting.
I20240503 07:58:30.923749 3274 tablet_bootstrap.cc:654] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:30.924015 2263 catalog_manager.cc:5484] T b46fc192e7aa439d988ee7c4e51603c2 P 80435ecadfcf4ede849c7a97abea952d reported cstate change: term changed from 0 to 1, leader changed from <none> to 80435ecadfcf4ede849c7a97abea952d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80435ecadfcf4ede849c7a97abea952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } }
I20240503 07:58:30.924988 2264 catalog_manager.cc:5484] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9405f157903141179a86c810396a416b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } }
I20240503 07:58:30.925119 3274 tablet_bootstrap.cc:492] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b: No bootstrap required, opened a new log
I20240503 07:58:30.925204 3274 ts_tablet_manager.cc:1367] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:58:30.925798 3274 raft_consensus.cc:348] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:30.925899 3274 raft_consensus.cc:374] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:30.925930 3274 raft_consensus.cc:729] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Initialized, Role: FOLLOWER
I20240503 07:58:30.926045 3274 consensus_queue.cc:260] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:30.926117 3274 raft_consensus.cc:388] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:30.926139 3274 raft_consensus.cc:482] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:30.926165 3274 raft_consensus.cc:3037] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:30.927163 3274 raft_consensus.cc:504] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:30.927299 3274 leader_election.cc:302] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:30.927394 3274 leader_election.cc:288] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:30.927604 3274 ts_tablet_manager.cc:1398] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:30.927729 3282 raft_consensus.cc:2781] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:30.927832 3282 raft_consensus.cc:686] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Leader. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:30.927937 3282 consensus_queue.cc:237] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:30.929517 2264 catalog_manager.cc:5484] T d81db8a9c3da42e7a26ef1c665b6c93b P 9405f157903141179a86c810396a416b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9405f157903141179a86c810396a416b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } }
W20240503 07:58:31.046921 3291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:31.047215 3291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:31.050460 3291 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 64.2985 ms
time per row: 0.0321493 ms
I20240503 07:58:31.165890 2263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49024:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:31.171658 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 403bf0824c2245a19f125db609649afd (DEFAULT_TABLE table=rebalance_test_table_1 [id=078c8b9967d94522a24e2b3bf16902cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:31.171689 2516 tablet_service.cc:1467] Processing CreateTablet for tablet e942e5928aab48eb9944179c292b8d1b (DEFAULT_TABLE table=rebalance_test_table_1 [id=078c8b9967d94522a24e2b3bf16902cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:31.171727 2385 tablet_service.cc:1467] Processing CreateTablet for tablet 527a672a8bd44251a56c5e47cc068e93 (DEFAULT_TABLE table=rebalance_test_table_1 [id=078c8b9967d94522a24e2b3bf16902cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:31.172047 2385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527a672a8bd44251a56c5e47cc068e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:31.172047 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e942e5928aab48eb9944179c292b8d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:31.174273 3275 tablet_bootstrap.cc:492] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d: Bootstrap starting.
I20240503 07:58:31.174482 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403bf0824c2245a19f125db609649afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:31.175179 3275 tablet_bootstrap.cc:654] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:31.175735 3274 tablet_bootstrap.cc:492] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b: Bootstrap starting.
I20240503 07:58:31.176257 3275 tablet_bootstrap.cc:492] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d: No bootstrap required, opened a new log
I20240503 07:58:31.176337 3275 ts_tablet_manager.cc:1367] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:31.176488 3274 tablet_bootstrap.cc:654] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:31.176846 3275 raft_consensus.cc:348] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.176939 3275 raft_consensus.cc:374] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:31.176965 3275 raft_consensus.cc:729] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Initialized, Role: FOLLOWER
I20240503 07:58:31.177098 3275 consensus_queue.cc:260] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.177174 3275 raft_consensus.cc:388] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:31.177210 3275 raft_consensus.cc:482] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:31.177246 3275 raft_consensus.cc:3037] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:31.177445 3274 tablet_bootstrap.cc:492] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b: No bootstrap required, opened a new log
I20240503 07:58:31.177517 3274 ts_tablet_manager.cc:1367] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:31.177927 3274 raft_consensus.cc:348] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.178016 3274 raft_consensus.cc:374] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:31.178045 3274 raft_consensus.cc:729] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Initialized, Role: FOLLOWER
I20240503 07:58:31.178018 3275 raft_consensus.cc:504] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.178136 3275 leader_election.cc:302] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80435ecadfcf4ede849c7a97abea952d; no voters:
I20240503 07:58:31.178154 3274 consensus_queue.cc:260] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.178213 3274 raft_consensus.cc:388] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:31.178221 3275 leader_election.cc:288] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:31.178239 3274 raft_consensus.cc:482] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:31.178275 3274 raft_consensus.cc:3037] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:31.178303 3280 raft_consensus.cc:2781] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:31.178404 3280 raft_consensus.cc:686] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Becoming Leader. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Running, Role: LEADER
I20240503 07:58:31.178526 3280 consensus_queue.cc:237] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.178627 3275 ts_tablet_manager.cc:1398] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:31.178761 3275 tablet_bootstrap.cc:492] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d: Bootstrap starting.
I20240503 07:58:31.179183 3274 raft_consensus.cc:504] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.179316 3274 leader_election.cc:302] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:31.179425 3274 leader_election.cc:288] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:31.179466 3282 raft_consensus.cc:2781] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:31.179713 3274 ts_tablet_manager.cc:1398] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:58:31.179971 3275 tablet_bootstrap.cc:654] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:31.180014 2263 catalog_manager.cc:5484] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d reported cstate change: term changed from 0 to 1, leader changed from <none> to 80435ecadfcf4ede849c7a97abea952d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80435ecadfcf4ede849c7a97abea952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } }
I20240503 07:58:31.180658 3282 raft_consensus.cc:686] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Leader. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:31.180800 3282 consensus_queue.cc:237] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.181028 3275 tablet_bootstrap.cc:492] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d: No bootstrap required, opened a new log
I20240503 07:58:31.181160 3275 ts_tablet_manager.cc:1367] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:31.181818 3275 raft_consensus.cc:348] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.181962 3275 raft_consensus.cc:374] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:31.182039 3275 raft_consensus.cc:729] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Initialized, Role: FOLLOWER
I20240503 07:58:31.182190 3275 consensus_queue.cc:260] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.182287 3275 raft_consensus.cc:388] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:31.182320 3275 raft_consensus.cc:482] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:31.182394 3275 raft_consensus.cc:3037] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:31.182364 2262 catalog_manager.cc:5484] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9405f157903141179a86c810396a416b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } }
I20240503 07:58:31.183243 3275 raft_consensus.cc:504] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.183370 3275 leader_election.cc:302] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80435ecadfcf4ede849c7a97abea952d; no voters:
I20240503 07:58:31.183473 3275 leader_election.cc:288] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:31.183521 3280 raft_consensus.cc:2781] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:31.183600 3280 raft_consensus.cc:686] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Becoming Leader. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Running, Role: LEADER
I20240503 07:58:31.183686 3275 ts_tablet_manager.cc:1398] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:31.183697 3280 consensus_queue.cc:237] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.185370 2262 catalog_manager.cc:5484] T 403bf0824c2245a19f125db609649afd P 80435ecadfcf4ede849c7a97abea952d reported cstate change: term changed from 0 to 1, leader changed from <none> to 80435ecadfcf4ede849c7a97abea952d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80435ecadfcf4ede849c7a97abea952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } }
W20240503 07:58:31.302297 3333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:31.302636 3333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:31.305876 3333 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 75.8068 ms
time per row: 0.0379034 ms
I20240503 07:58:31.425729 2261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49024:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:31.431974 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 9841efae7c614c8d826151a88a1c5c6f (DEFAULT_TABLE table=rebalance_test_table_2 [id=1bdfc2dd24fe4ed6bae2f15170340978]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:31.431968 2386 tablet_service.cc:1467] Processing CreateTablet for tablet 5031a2c9d5af4289928e0e43488931f8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1bdfc2dd24fe4ed6bae2f15170340978]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:31.432325 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5031a2c9d5af4289928e0e43488931f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:31.432334 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9841efae7c614c8d826151a88a1c5c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:31.434096 3275 tablet_bootstrap.cc:492] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d: Bootstrap starting.
I20240503 07:58:31.431968 2385 tablet_service.cc:1467] Processing CreateTablet for tablet e9e176777a54493f94cdf609210874b6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1bdfc2dd24fe4ed6bae2f15170340978]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:31.434336 3274 tablet_bootstrap.cc:492] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b: Bootstrap starting.
I20240503 07:58:31.434365 2385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e176777a54493f94cdf609210874b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:31.435086 3275 tablet_bootstrap.cc:654] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:31.435295 3274 tablet_bootstrap.cc:654] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:31.436203 3275 tablet_bootstrap.cc:492] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d: No bootstrap required, opened a new log
I20240503 07:58:31.436295 3275 ts_tablet_manager.cc:1367] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:31.436333 3274 tablet_bootstrap.cc:492] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b: No bootstrap required, opened a new log
I20240503 07:58:31.436407 3274 ts_tablet_manager.cc:1367] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:58:31.436749 3275 raft_consensus.cc:348] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.436834 3275 raft_consensus.cc:374] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:31.436784 3274 raft_consensus.cc:348] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.436854 3275 raft_consensus.cc:729] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Initialized, Role: FOLLOWER
I20240503 07:58:31.436864 3274 raft_consensus.cc:374] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:31.436885 3274 raft_consensus.cc:729] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Initialized, Role: FOLLOWER
I20240503 07:58:31.436968 3275 consensus_queue.cc:260] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.436990 3274 consensus_queue.cc:260] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.437031 3275 raft_consensus.cc:388] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:31.437052 3274 raft_consensus.cc:388] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:31.437054 3275 raft_consensus.cc:482] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:31.437074 3274 raft_consensus.cc:482] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:31.437083 3275 raft_consensus.cc:3037] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:31.437105 3274 raft_consensus.cc:3037] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:31.437727 3274 raft_consensus.cc:504] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.437765 3275 raft_consensus.cc:504] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.437860 3274 leader_election.cc:302] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:31.437882 3275 leader_election.cc:302] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80435ecadfcf4ede849c7a97abea952d; no voters:
I20240503 07:58:31.437933 3274 leader_election.cc:288] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:31.437968 3275 leader_election.cc:288] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:31.438009 3281 raft_consensus.cc:2781] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:31.438151 3274 ts_tablet_manager.cc:1398] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:58:31.438172 3275 ts_tablet_manager.cc:1398] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:31.438176 3281 raft_consensus.cc:686] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Leader. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:31.438319 3280 raft_consensus.cc:2781] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:31.438294 3281 consensus_queue.cc:237] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.438426 3280 raft_consensus.cc:686] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Becoming Leader. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Running, Role: LEADER
I20240503 07:58:31.438561 3280 consensus_queue.cc:237] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.438310 3274 tablet_bootstrap.cc:492] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b: Bootstrap starting.
I20240503 07:58:31.440076 3274 tablet_bootstrap.cc:654] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:31.440100 2260 catalog_manager.cc:5484] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d reported cstate change: term changed from 0 to 1, leader changed from <none> to 80435ecadfcf4ede849c7a97abea952d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80435ecadfcf4ede849c7a97abea952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } }
I20240503 07:58:31.440100 2261 catalog_manager.cc:5484] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9405f157903141179a86c810396a416b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } }
I20240503 07:58:31.441133 3274 tablet_bootstrap.cc:492] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b: No bootstrap required, opened a new log
I20240503 07:58:31.441256 3274 ts_tablet_manager.cc:1367] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:31.441936 3274 raft_consensus.cc:348] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.442096 3274 raft_consensus.cc:374] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:31.442174 3274 raft_consensus.cc:729] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Initialized, Role: FOLLOWER
I20240503 07:58:31.442773 3274 consensus_queue.cc:260] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.443040 3274 raft_consensus.cc:388] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:31.443115 3274 raft_consensus.cc:482] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:31.443166 3274 raft_consensus.cc:3037] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:31.443965 3274 raft_consensus.cc:504] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.444110 3274 leader_election.cc:302] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:31.444254 3363 raft_consensus.cc:2781] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:31.444337 3363 raft_consensus.cc:686] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Leader. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:31.444453 3363 consensus_queue.cc:237] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.444837 3274 leader_election.cc:288] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:31.445143 3274 ts_tablet_manager.cc:1398] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:58:31.445863 2261 catalog_manager.cc:5484] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9405f157903141179a86c810396a416b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } }
W20240503 07:58:31.565485 3369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:31.565780 3369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:31.569022 3369 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 80.2848 ms
time per row: 0.0401424 ms
I20240503 07:58:31.692133 2260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49024:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:31.698024 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed9251d58be42d3841e8eb72b520353 (DEFAULT_TABLE table=rebalance_test_table_3 [id=28225e90d3cd4d9e9330eaeb67f6f2a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:31.698081 2385 tablet_service.cc:1467] Processing CreateTablet for tablet d30b596efc8f4f2190a6ca4eba17640b (DEFAULT_TABLE table=rebalance_test_table_3 [id=28225e90d3cd4d9e9330eaeb67f6f2a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:31.698024 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 12be85cd12b84b4aa6389177679cd9d3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=28225e90d3cd4d9e9330eaeb67f6f2a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:31.698460 2385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30b596efc8f4f2190a6ca4eba17640b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:31.698493 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed9251d58be42d3841e8eb72b520353. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:31.698655 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12be85cd12b84b4aa6389177679cd9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:31.700256 3275 tablet_bootstrap.cc:492] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d: Bootstrap starting.
I20240503 07:58:31.700560 3274 tablet_bootstrap.cc:492] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b: Bootstrap starting.
I20240503 07:58:31.701180 3275 tablet_bootstrap.cc:654] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:31.701246 3274 tablet_bootstrap.cc:654] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:31.702107 3275 tablet_bootstrap.cc:492] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d: No bootstrap required, opened a new log
I20240503 07:58:31.702138 3274 tablet_bootstrap.cc:492] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b: No bootstrap required, opened a new log
I20240503 07:58:31.702183 3275 ts_tablet_manager.cc:1367] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:31.702209 3274 ts_tablet_manager.cc:1367] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:31.702663 3275 raft_consensus.cc:348] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.702679 3274 raft_consensus.cc:348] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.702771 3274 raft_consensus.cc:374] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:31.702771 3275 raft_consensus.cc:374] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:31.702802 3274 raft_consensus.cc:729] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Initialized, Role: FOLLOWER
I20240503 07:58:31.702808 3275 raft_consensus.cc:729] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Initialized, Role: FOLLOWER
I20240503 07:58:31.702932 3275 consensus_queue.cc:260] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.702934 3274 consensus_queue.cc:260] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.702991 3275 raft_consensus.cc:388] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:31.703007 3274 raft_consensus.cc:388] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:31.703011 3275 raft_consensus.cc:482] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:31.703029 3274 raft_consensus.cc:482] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:31.703040 3275 raft_consensus.cc:3037] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:31.703058 3274 raft_consensus.cc:3037] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:31.703707 3274 raft_consensus.cc:504] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.703698 3275 raft_consensus.cc:504] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.703805 3274 leader_election.cc:302] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:31.703810 3275 leader_election.cc:302] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80435ecadfcf4ede849c7a97abea952d; no voters:
I20240503 07:58:31.703874 3274 leader_election.cc:288] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:31.703883 3275 leader_election.cc:288] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:31.703933 3363 raft_consensus.cc:2781] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:31.703941 3280 raft_consensus.cc:2781] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:31.704205 3363 raft_consensus.cc:686] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Leader. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:31.704269 3274 ts_tablet_manager.cc:1398] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:31.704380 3363 consensus_queue.cc:237] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.704563 3280 raft_consensus.cc:686] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Becoming Leader. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Running, Role: LEADER
I20240503 07:58:31.704685 3280 consensus_queue.cc:237] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.705224 3275 ts_tablet_manager.cc:1398] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:31.705425 3275 tablet_bootstrap.cc:492] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d: Bootstrap starting.
I20240503 07:58:31.706241 2260 catalog_manager.cc:5484] T d30b596efc8f4f2190a6ca4eba17640b P 9405f157903141179a86c810396a416b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9405f157903141179a86c810396a416b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } }
I20240503 07:58:31.706373 2255 catalog_manager.cc:5484] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d reported cstate change: term changed from 0 to 1, leader changed from <none> to 80435ecadfcf4ede849c7a97abea952d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80435ecadfcf4ede849c7a97abea952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } }
I20240503 07:58:31.706585 3275 tablet_bootstrap.cc:654] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:31.707850 3275 tablet_bootstrap.cc:492] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d: No bootstrap required, opened a new log
I20240503 07:58:31.707932 3275 ts_tablet_manager.cc:1367] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:31.708361 3275 raft_consensus.cc:348] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.708462 3275 raft_consensus.cc:374] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:31.708549 3275 raft_consensus.cc:729] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Initialized, Role: FOLLOWER
I20240503 07:58:31.708696 3275 consensus_queue.cc:260] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.708842 3275 raft_consensus.cc:388] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:31.708890 3275 raft_consensus.cc:482] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:31.708963 3275 raft_consensus.cc:3037] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:31.709915 3275 raft_consensus.cc:504] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.710060 3275 leader_election.cc:302] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80435ecadfcf4ede849c7a97abea952d; no voters:
I20240503 07:58:31.710158 3275 leader_election.cc:288] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:31.710212 3280 raft_consensus.cc:2781] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:31.710317 3280 raft_consensus.cc:686] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Becoming Leader. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Running, Role: LEADER
I20240503 07:58:31.710542 3280 consensus_queue.cc:237] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.710593 3275 ts_tablet_manager.cc:1398] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:31.712996 2255 catalog_manager.cc:5484] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d reported cstate change: term changed from 0 to 1, leader changed from <none> to 80435ecadfcf4ede849c7a97abea952d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80435ecadfcf4ede849c7a97abea952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } }
W20240503 07:58:31.829543 3403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:31.829849 3403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:31.833182 3403 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 73.6952 ms
time per row: 0.0368476 ms
I20240503 07:58:31.950047 2259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49024:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:31.956009 2385 tablet_service.cc:1467] Processing CreateTablet for tablet dea872e4b037445c93067135349f57f4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=76719076b11844e4b721cc235f022423]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:31.956377 2385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea872e4b037445c93067135349f57f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:31.956418 2386 tablet_service.cc:1467] Processing CreateTablet for tablet ef2c306c822246ef949e287a9937d94d (DEFAULT_TABLE table=rebalance_test_table_4 [id=76719076b11844e4b721cc235f022423]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:31.956646 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2c306c822246ef949e287a9937d94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:31.958482 3274 tablet_bootstrap.cc:492] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b: Bootstrap starting.
I20240503 07:58:31.958595 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 52c02ae315844d90aa9454b9c022f27d (DEFAULT_TABLE table=rebalance_test_table_4 [id=76719076b11844e4b721cc235f022423]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:31.958854 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c02ae315844d90aa9454b9c022f27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:31.959414 3274 tablet_bootstrap.cc:654] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:31.960507 3274 tablet_bootstrap.cc:492] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b: No bootstrap required, opened a new log
I20240503 07:58:31.960603 3274 ts_tablet_manager.cc:1367] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:31.961010 3275 tablet_bootstrap.cc:492] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d: Bootstrap starting.
I20240503 07:58:31.961071 3274 raft_consensus.cc:348] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.961171 3274 raft_consensus.cc:374] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:31.961200 3274 raft_consensus.cc:729] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Initialized, Role: FOLLOWER
I20240503 07:58:31.961328 3274 consensus_queue.cc:260] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.961418 3274 raft_consensus.cc:388] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:31.961448 3274 raft_consensus.cc:482] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:31.961483 3274 raft_consensus.cc:3037] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:31.961850 3275 tablet_bootstrap.cc:654] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:31.962074 3274 raft_consensus.cc:504] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.962177 3274 leader_election.cc:302] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:31.962246 3274 leader_election.cc:288] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:31.962330 3281 raft_consensus.cc:2781] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:31.962497 3274 ts_tablet_manager.cc:1398] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:31.962713 3281 raft_consensus.cc:686] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Leader. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:31.962805 3274 tablet_bootstrap.cc:492] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b: Bootstrap starting.
I20240503 07:58:31.962836 3281 consensus_queue.cc:237] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.962955 3275 tablet_bootstrap.cc:492] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d: No bootstrap required, opened a new log
I20240503 07:58:31.963027 3275 ts_tablet_manager.cc:1367] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:31.963465 3275 raft_consensus.cc:348] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.963567 3275 raft_consensus.cc:374] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:31.963593 3274 tablet_bootstrap.cc:654] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:31.963598 3275 raft_consensus.cc:729] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Initialized, Role: FOLLOWER
I20240503 07:58:31.963716 3275 consensus_queue.cc:260] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.963794 3275 raft_consensus.cc:388] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:31.963821 3275 raft_consensus.cc:482] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:31.963855 3275 raft_consensus.cc:3037] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:31.964465 2257 catalog_manager.cc:5484] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9405f157903141179a86c810396a416b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } }
I20240503 07:58:31.964659 3274 tablet_bootstrap.cc:492] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b: No bootstrap required, opened a new log
I20240503 07:58:31.964730 3274 ts_tablet_manager.cc:1367] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:31.964789 3275 raft_consensus.cc:504] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.964912 3275 leader_election.cc:302] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80435ecadfcf4ede849c7a97abea952d; no voters:
I20240503 07:58:31.964990 3275 leader_election.cc:288] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:31.965050 3280 raft_consensus.cc:2781] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:31.965132 3280 raft_consensus.cc:686] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Becoming Leader. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Running, Role: LEADER
I20240503 07:58:31.965159 3274 raft_consensus.cc:348] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.965206 3275 ts_tablet_manager.cc:1398] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:31.965234 3274 raft_consensus.cc:374] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:31.965265 3274 raft_consensus.cc:729] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Initialized, Role: FOLLOWER
I20240503 07:58:31.965237 3280 consensus_queue.cc:237] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:31.965373 3274 consensus_queue.cc:260] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.965451 3274 raft_consensus.cc:388] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:31.965476 3274 raft_consensus.cc:482] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:31.965509 3274 raft_consensus.cc:3037] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:31.966185 3274 raft_consensus.cc:504] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.966298 3274 leader_election.cc:302] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:31.966569 3274 leader_election.cc:288] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:31.966795 3274 ts_tablet_manager.cc:1398] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:31.966779 2259 catalog_manager.cc:5484] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d reported cstate change: term changed from 0 to 1, leader changed from <none> to 80435ecadfcf4ede849c7a97abea952d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80435ecadfcf4ede849c7a97abea952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } }
I20240503 07:58:31.966920 3281 raft_consensus.cc:2781] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:31.967001 3281 raft_consensus.cc:686] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Leader. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:31.967099 3281 consensus_queue.cc:237] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:31.969069 2257 catalog_manager.cc:5484] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9405f157903141179a86c810396a416b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } }
W20240503 07:58:32.085956 3435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:32.086266 3435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:32.089506 3435 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 86.9716 ms
time per row: 0.0434858 ms
I20240503 07:58:32.219681 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:38167
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:34437
--webserver_port=33621
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 07:58:32.331861 3467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:32.332113 3467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:32.332157 3467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:32.335439 3467 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:32.335516 3467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:32.335602 3467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:58:32.339512 3467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:38167
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=33621
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.3467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:32.340502 3467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:32.341771 3467 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:32.350724 3473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.351295 3472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.351876 3475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:32.352355 3467 server_base.cc:1034] running on GCE node
I20240503 07:58:32.352806 3467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:32.353432 3467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:32.354610 3467 hybrid_clock.cc:648] HybridClock initialized: now 1714723112354594 us; error 31 us; skew 500 ppm
I20240503 07:58:32.356737 3467 webserver.cc:460] Webserver started at http://127.0.99.131:33621/ using document root <none> and password file <none>
I20240503 07:58:32.357340 3467 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:32.357414 3467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:32.361039 3467 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:32.363246 3481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.364404 3467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 07:58:32.364522 3467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e4a21231285448c88ef7465d265f20b5"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:32.365031 3467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:32.396576 3467 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:32.397352 3467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:32.397523 3467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:32.398135 3467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:32.399391 3488 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 07:58:32.402935 3467 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 07:58:32.403012 3467 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20240503 07:58:32.403084 3467 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 07:58:32.404711 3467 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 07:58:32.404758 3467 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240503 07:58:32.405104 3488 tablet_bootstrap.cc:492] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5: Bootstrap starting.
I20240503 07:58:32.414861 3467 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:38167
I20240503 07:58:32.414886 3595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:38167 every 8 connection(s)
I20240503 07:58:32.416062 3467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:58:32.419590 3488 log.cc:824] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:32.422422 3488 tablet_bootstrap.cc:492] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:32.422786 3488 tablet_bootstrap.cc:492] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5: Bootstrap complete.
I20240503 07:58:32.423781 3488 ts_tablet_manager.cc:1367] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20240503 07:58:32.425892 3596 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:32.426050 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 3467
I20240503 07:58:32.426151 3596 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:32.426566 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:46551
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:34437
--webserver_port=45215
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:32.426822 3596 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:32.427083 3488 raft_consensus.cc:348] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:32.427518 3488 raft_consensus.cc:729] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a21231285448c88ef7465d265f20b5, State: Initialized, Role: FOLLOWER
I20240503 07:58:32.428048 3488 consensus_queue.cc:260] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:32.428220 3488 raft_consensus.cc:388] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:32.428339 3488 raft_consensus.cc:482] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:32.428458 3488 raft_consensus.cc:3037] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:32.430222 3488 raft_consensus.cc:504] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:32.430613 3488 leader_election.cc:302] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4a21231285448c88ef7465d265f20b5; no voters:
I20240503 07:58:32.430652 2261 ts_manager.cc:194] Re-registered known tserver with Master: e4a21231285448c88ef7465d265f20b5 (127.0.99.131:38167)
I20240503 07:58:32.430934 3488 leader_election.cc:288] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 07:58:32.431733 3488 ts_tablet_manager.cc:1398] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20240503 07:58:32.431797 2261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:58313
I20240503 07:58:32.432386 3601 raft_consensus.cc:2781] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:32.432559 3601 raft_consensus.cc:686] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [term 2 LEADER]: Becoming Leader. State: Replica: e4a21231285448c88ef7465d265f20b5, State: Running, Role: LEADER
I20240503 07:58:32.432857 3601 consensus_queue.cc:237] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:32.434605 3596 heartbeater.cc:503] Master 127.0.99.190:34437 was elected leader, sending a full tablet report...
I20240503 07:58:32.436084 2261 catalog_manager.cc:5484] T 0189e0b9411f41908cbce67fce3ea308 P e4a21231285448c88ef7465d265f20b5 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "e4a21231285448c88ef7465d265f20b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } } }
W20240503 07:58:32.547618 3600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:32.547865 3600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:32.547914 3600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:32.551165 3600 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:32.551250 3600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:32.551328 3600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:58:32.555209 3600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:46551
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=45215
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.3600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:32.556238 3600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:32.557437 3600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:32.566325 3613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.566805 3610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.567293 3611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:32.567885 3600 server_base.cc:1034] running on GCE node
I20240503 07:58:32.568315 3600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:32.568997 3600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:32.570151 3600 hybrid_clock.cc:648] HybridClock initialized: now 1714723112570136 us; error 35 us; skew 500 ppm
I20240503 07:58:32.572243 3600 webserver.cc:460] Webserver started at http://127.0.99.132:45215/ using document root <none> and password file <none>
I20240503 07:58:32.572901 3600 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:32.572980 3600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:32.576762 3600 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240503 07:58:32.578835 3619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.579958 3600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:32.580066 3600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1147cda1821744448cfea2164c8f3d58"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:32.580529 3600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:32.600776 3600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:32.601538 3600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:32.601712 3600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:32.602298 3600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:32.603291 3600 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:32.603341 3600 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.603405 3600 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:32.603431 3600 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.612200 3600 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:46551
I20240503 07:58:32.612226 3732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:46551 every 8 connection(s)
I20240503 07:58:32.613324 3600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:58:32.614447 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 3600
I20240503 07:58:32.614812 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:43115
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:34437
--webserver_port=40781
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:32.623898 3733 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:32.624154 3733 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:32.624676 3733 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:32.625743 2261 ts_manager.cc:194] Re-registered known tserver with Master: 1147cda1821744448cfea2164c8f3d58 (127.0.99.132:46551)
I20240503 07:58:32.626612 2261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:46739
W20240503 07:58:32.741261 3736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:32.741528 3736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:32.741587 3736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:32.745066 3736 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:32.745154 3736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:32.745229 3736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:58:32.749238 3736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:43115
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=40781
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.3736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:32.750270 3736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:32.751613 3736 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:32.760718 3744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.761278 3741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.761821 3742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:32.762468 3736 server_base.cc:1034] running on GCE node
I20240503 07:58:32.762894 3736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:32.763547 3736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:32.764708 3736 hybrid_clock.cc:648] HybridClock initialized: now 1714723112764694 us; error 29 us; skew 500 ppm
I20240503 07:58:32.766867 3736 webserver.cc:460] Webserver started at http://127.0.99.133:40781/ using document root <none> and password file <none>
I20240503 07:58:32.767478 3736 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:32.767551 3736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:32.771378 3736 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:32.773712 3750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.774901 3736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:32.775020 3736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6fa4f16dd2a545dbb01a0b260ce68744"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:32.775497 3736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:32.789158 3736 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:32.789891 3736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:32.790058 3736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:32.790695 3736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:32.791672 3736 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:32.791723 3736 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.791775 3736 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:32.791802 3736 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.800209 3736 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:43115
I20240503 07:58:32.800242 3863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:43115 every 8 connection(s)
I20240503 07:58:32.801350 3736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:58:32.801501 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 3736
I20240503 07:58:32.801939 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:46183
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:34437
--webserver_port=36101
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:32.811411 3864 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:32.811704 3864 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:32.812309 3864 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:32.813515 2261 ts_manager.cc:194] Re-registered known tserver with Master: 6fa4f16dd2a545dbb01a0b260ce68744 (127.0.99.133:43115)
I20240503 07:58:32.814152 2261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:49737
W20240503 07:58:32.922061 3866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:32.922312 3866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:32.922412 3866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:32.925719 3866 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:32.925806 3866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:32.925907 3866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:58:32.929926 3866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:46183
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=36101
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.3866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:32.931022 3866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:32.932282 3866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:32.941305 3875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.941829 3873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.942301 3872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:32.942919 3866 server_base.cc:1034] running on GCE node
I20240503 07:58:32.943383 3866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:32.944057 3866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:32.945207 3866 hybrid_clock.cc:648] HybridClock initialized: now 1714723112945183 us; error 46 us; skew 500 ppm
I20240503 07:58:32.947302 3866 webserver.cc:460] Webserver started at http://127.0.99.134:36101/ using document root <none> and password file <none>
I20240503 07:58:32.947944 3866 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:32.948025 3866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:32.952025 3866 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20240503 07:58:32.954035 3881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.955188 3866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:32.955318 3866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6245203dd9bb46a18e8f3b4526d16b82"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:32.955830 3866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:32.977922 3866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:32.978729 3866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:32.978904 3866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:32.979593 3866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:32.980648 3866 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:32.980701 3866 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.980759 3866 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:32.980787 3866 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.990036 3866 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:46183
I20240503 07:58:32.990058 3994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:46183 every 8 connection(s)
I20240503 07:58:32.991253 3866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:58:32.998648 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 3866
I20240503 07:58:32.999118 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:34969
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:34437
--webserver_port=37461
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:33.001065 3995 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:33.001369 3995 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:33.001896 3995 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:33.003083 2261 ts_manager.cc:194] Re-registered known tserver with Master: 6245203dd9bb46a18e8f3b4526d16b82 (127.0.99.134:46183)
I20240503 07:58:33.003865 2261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:53461
W20240503 07:58:33.117058 3998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:33.117306 3998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:33.117350 3998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:33.120544 3998 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:33.120625 3998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:33.120697 3998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:58:33.124575 3998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:34969
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=37461
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.3998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:33.125564 3998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:33.126883 3998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:33.135622 4004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:33.136103 4006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:33.136633 4003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:33.137171 3998 server_base.cc:1034] running on GCE node
I20240503 07:58:33.137617 3998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:33.138259 3998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:33.139413 3998 hybrid_clock.cc:648] HybridClock initialized: now 1714723113139391 us; error 37 us; skew 500 ppm
I20240503 07:58:33.141391 3998 webserver.cc:460] Webserver started at http://127.0.99.135:37461/ using document root <none> and password file <none>
I20240503 07:58:33.142032 3998 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:33.142118 3998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:33.145874 3998 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:33.147964 4012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:33.149091 3998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:33.149214 3998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "27715b19d4ad40d89c2d12e66aca6a34"
format_stamp: "Formatted at 2024-05-03 07:58:26 on dist-test-slave-bj8b"
I20240503 07:58:33.149685 3998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:33.168845 3998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:33.169705 3998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:33.169885 3998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:33.170504 3998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:33.171494 3998 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:33.171546 3998 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:33.171600 3998 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:33.171624 3998 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:33.180953 3998 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:34969
I20240503 07:58:33.180984 4125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:34969 every 8 connection(s)
I20240503 07:58:33.182121 3998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:58:33.186424 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 3998
I20240503 07:58:33.192054 4126 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:33.192368 4126 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:33.193035 4126 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:33.194293 2261 ts_manager.cc:194] Re-registered known tserver with Master: 27715b19d4ad40d89c2d12e66aca6a34 (127.0.99.135:34969)
I20240503 07:58:33.195127 2261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:39739
I20240503 07:58:33.404285 2385 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:58:33.404366 3667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.406450 3929 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.408557 2517 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:58:33.408854 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.416857 3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.427313 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:33.541255 3667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.543038 3929 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.548748 3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.550771 2385 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:58:33.556799 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.562803 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:33.563956 2517 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:58:33.613795 2406 consensus_queue.cc:237] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } }
W20240503 07:58:33.620672 2340 consensus_peers.cc:485] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b -> Peer 1147cda1821744448cfea2164c8f3d58 (127.0.99.132:46551): Couldn't send request to peer 1147cda1821744448cfea2164c8f3d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dea872e4b037445c93067135349f57f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:33.628794 3733 heartbeater.cc:503] Master 127.0.99.190:34437 was elected leader, sending a full tablet report...
I20240503 07:58:33.816390 3864 heartbeater.cc:503] Master 127.0.99.190:34437 was elected leader, sending a full tablet report...
I20240503 07:58:34.006209 3995 heartbeater.cc:503] Master 127.0.99.190:34437 was elected leader, sending a full tablet report...
I20240503 07:58:34.180274 4229 ts_tablet_manager.cc:902] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58: Initiating tablet copy from peer 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:34.181155 4229 tablet_copy_client.cc:308] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35381
I20240503 07:58:34.185639 2426 tablet_copy_service.cc:140] P 9405f157903141179a86c810396a416b: Received BeginTabletCopySession request for tablet dea872e4b037445c93067135349f57f4 from peer 1147cda1821744448cfea2164c8f3d58 ({username='slave'} at 127.0.99.132:51409)
I20240503 07:58:34.185822 2426 tablet_copy_service.cc:161] P 9405f157903141179a86c810396a416b: Beginning new tablet copy session on tablet dea872e4b037445c93067135349f57f4 from peer 1147cda1821744448cfea2164c8f3d58 at {username='slave'} at 127.0.99.132:51409: session id = 1147cda1821744448cfea2164c8f3d58-dea872e4b037445c93067135349f57f4
I20240503 07:58:34.187158 2426 tablet_copy_source_session.cc:205] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:34.190987 4229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea872e4b037445c93067135349f57f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.196800 4229 tablet_copy_client.cc:783] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:34.196996 4229 tablet_copy_client.cc:647] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:34.197937 4126 heartbeater.cc:503] Master 127.0.99.190:34437 was elected leader, sending a full tablet report...
I20240503 07:58:34.199362 4229 tablet_copy_client.cc:523] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:34.201200 4229 tablet_bootstrap.cc:492] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58: Bootstrap starting.
I20240503 07:58:34.207039 4229 log.cc:824] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:34.228935 4229 tablet_bootstrap.cc:492] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:58:34.229395 4229 tablet_bootstrap.cc:492] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58: Bootstrap complete.
I20240503 07:58:34.229736 4229 ts_tablet_manager.cc:1367] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.000s
I20240503 07:58:34.232262 4229 raft_consensus.cc:348] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:34.232928 4229 raft_consensus.cc:729] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1147cda1821744448cfea2164c8f3d58, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.233273 4229 consensus_queue.cc:260] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } }
I20240503 07:58:34.234153 4229 ts_tablet_manager.cc:1398] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20240503 07:58:34.235150 2426 tablet_copy_service.cc:342] P 9405f157903141179a86c810396a416b: Request end of tablet copy session 1147cda1821744448cfea2164c8f3d58-dea872e4b037445c93067135349f57f4 received from {username='slave'} at 127.0.99.132:51409
I20240503 07:58:34.235263 2426 tablet_copy_service.cc:433] P 9405f157903141179a86c810396a416b: ending tablet copy session 1147cda1821744448cfea2164c8f3d58-dea872e4b037445c93067135349f57f4 on tablet dea872e4b037445c93067135349f57f4 with peer 1147cda1821744448cfea2164c8f3d58
I20240503 07:58:34.676054 3687 raft_consensus.cc:1203] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:34.677810 4238 raft_consensus.cc:2932] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1147cda1821744448cfea2164c8f3d58 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } } }
I20240503 07:58:34.678311 3687 raft_consensus.cc:2932] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1147cda1821744448cfea2164c8f3d58 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } } }
I20240503 07:58:34.679421 4238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:39842 (request call id 0) took 1066 ms. Trace:
I20240503 07:58:34.679543 4238 rpcz_store.cc:276] 0503 07:58:33.612977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 07:58:33.613051 (+ 74us) service_pool.cc:225] Handling call
0503 07:58:33.615075 (+ 2024us) log.cc:842] Serialized 257 byte log entry
0503 07:58:34.679406 (+1064331us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":78,"raft.queue_time_us":88,"raft.run_cpu_time_us":383,"raft.run_wall_time_us":382,"spinlock_wait_cycles":11520,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":195,"wal-append.run_cpu_time_us":402,"wal-append.run_wall_time_us":1000487}
I20240503 07:58:34.681607 2260 catalog_manager.cc:5484] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b reported cstate change: config changed from index -1 to 4, VOTER 1147cda1821744448cfea2164c8f3d58 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } } }
I20240503 07:58:34.700706 2406 consensus_queue.cc:237] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } }
W20240503 07:58:34.705839 2342 consensus_peers.cc:485] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b -> Peer 6fa4f16dd2a545dbb01a0b260ce68744 (127.0.99.133:43115): Couldn't send request to peer 6fa4f16dd2a545dbb01a0b260ce68744. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5031a2c9d5af4289928e0e43488931f8. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:35.162707 4264 ts_tablet_manager.cc:902] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744: Initiating tablet copy from peer 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:35.163473 4264 tablet_copy_client.cc:308] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35381
I20240503 07:58:35.167076 2426 tablet_copy_service.cc:140] P 9405f157903141179a86c810396a416b: Received BeginTabletCopySession request for tablet 5031a2c9d5af4289928e0e43488931f8 from peer 6fa4f16dd2a545dbb01a0b260ce68744 ({username='slave'} at 127.0.99.133:43235)
I20240503 07:58:35.167204 2426 tablet_copy_service.cc:161] P 9405f157903141179a86c810396a416b: Beginning new tablet copy session on tablet 5031a2c9d5af4289928e0e43488931f8 from peer 6fa4f16dd2a545dbb01a0b260ce68744 at {username='slave'} at 127.0.99.133:43235: session id = 6fa4f16dd2a545dbb01a0b260ce68744-5031a2c9d5af4289928e0e43488931f8
I20240503 07:58:35.167966 2426 tablet_copy_source_session.cc:205] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:35.171375 4264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5031a2c9d5af4289928e0e43488931f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:35.177111 4264 tablet_copy_client.cc:783] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:35.177325 4264 tablet_copy_client.cc:647] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:35.178920 4264 tablet_copy_client.cc:523] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:35.180716 4264 tablet_bootstrap.cc:492] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744: Bootstrap starting.
I20240503 07:58:35.186945 4264 log.cc:824] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:35.207990 4264 tablet_bootstrap.cc:492] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:58:35.208424 4264 tablet_bootstrap.cc:492] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744: Bootstrap complete.
I20240503 07:58:35.208765 4264 ts_tablet_manager.cc:1367] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744: Time spent bootstrapping tablet: real 0.028s user 0.024s sys 0.003s
I20240503 07:58:35.211079 4264 raft_consensus.cc:348] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:35.211730 4264 raft_consensus.cc:729] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa4f16dd2a545dbb01a0b260ce68744, State: Initialized, Role: FOLLOWER
I20240503 07:58:35.212059 4264 consensus_queue.cc:260] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } }
I20240503 07:58:35.212884 4264 ts_tablet_manager.cc:1398] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240503 07:58:35.213685 2426 tablet_copy_service.cc:342] P 9405f157903141179a86c810396a416b: Request end of tablet copy session 6fa4f16dd2a545dbb01a0b260ce68744-5031a2c9d5af4289928e0e43488931f8 received from {username='slave'} at 127.0.99.133:43235
I20240503 07:58:35.213765 2426 tablet_copy_service.cc:433] P 9405f157903141179a86c810396a416b: ending tablet copy session 6fa4f16dd2a545dbb01a0b260ce68744-5031a2c9d5af4289928e0e43488931f8 on tablet 5031a2c9d5af4289928e0e43488931f8 with peer 6fa4f16dd2a545dbb01a0b260ce68744
I20240503 07:58:35.573802 3818 raft_consensus.cc:1203] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:35.575204 4272 raft_consensus.cc:2932] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6fa4f16dd2a545dbb01a0b260ce68744 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } } }
I20240503 07:58:35.575816 3818 raft_consensus.cc:2932] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6fa4f16dd2a545dbb01a0b260ce68744 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } } }
I20240503 07:58:35.579183 2260 catalog_manager.cc:5484] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b reported cstate change: config changed from index -1 to 4, VOTER 6fa4f16dd2a545dbb01a0b260ce68744 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } } }
I20240503 07:58:35.596426 2537 consensus_queue.cc:237] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } }
W20240503 07:58:35.602200 2473 consensus_peers.cc:485] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d -> Peer 6245203dd9bb46a18e8f3b4526d16b82 (127.0.99.134:46183): Couldn't send request to peer 6245203dd9bb46a18e8f3b4526d16b82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e942e5928aab48eb9944179c292b8d1b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:36.083021 4298 ts_tablet_manager.cc:902] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82: Initiating tablet copy from peer 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895)
I20240503 07:58:36.083819 4298 tablet_copy_client.cc:308] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33895
I20240503 07:58:36.087666 2557 tablet_copy_service.cc:140] P 80435ecadfcf4ede849c7a97abea952d: Received BeginTabletCopySession request for tablet e942e5928aab48eb9944179c292b8d1b from peer 6245203dd9bb46a18e8f3b4526d16b82 ({username='slave'} at 127.0.99.134:43697)
I20240503 07:58:36.087834 2557 tablet_copy_service.cc:161] P 80435ecadfcf4ede849c7a97abea952d: Beginning new tablet copy session on tablet e942e5928aab48eb9944179c292b8d1b from peer 6245203dd9bb46a18e8f3b4526d16b82 at {username='slave'} at 127.0.99.134:43697: session id = 6245203dd9bb46a18e8f3b4526d16b82-e942e5928aab48eb9944179c292b8d1b
I20240503 07:58:36.089090 2557 tablet_copy_source_session.cc:205] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:36.092752 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e942e5928aab48eb9944179c292b8d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:36.098461 4298 tablet_copy_client.cc:783] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:36.098666 4298 tablet_copy_client.cc:647] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:36.100481 4298 tablet_copy_client.cc:523] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:36.102274 4298 tablet_bootstrap.cc:492] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82: Bootstrap starting.
I20240503 07:58:36.107636 4298 log.cc:824] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:36.127231 4298 tablet_bootstrap.cc:492] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:58:36.127650 4298 tablet_bootstrap.cc:492] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82: Bootstrap complete.
I20240503 07:58:36.127971 4298 ts_tablet_manager.cc:1367] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82: Time spent bootstrapping tablet: real 0.026s user 0.026s sys 0.000s
I20240503 07:58:36.130115 4298 raft_consensus.cc:348] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:36.130765 4298 raft_consensus.cc:729] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6245203dd9bb46a18e8f3b4526d16b82, State: Initialized, Role: FOLLOWER
I20240503 07:58:36.131088 4298 consensus_queue.cc:260] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } }
I20240503 07:58:36.131919 4298 ts_tablet_manager.cc:1398] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 07:58:36.132833 2557 tablet_copy_service.cc:342] P 80435ecadfcf4ede849c7a97abea952d: Request end of tablet copy session 6245203dd9bb46a18e8f3b4526d16b82-e942e5928aab48eb9944179c292b8d1b received from {username='slave'} at 127.0.99.134:43697
I20240503 07:58:36.132933 2557 tablet_copy_service.cc:433] P 80435ecadfcf4ede849c7a97abea952d: ending tablet copy session 6245203dd9bb46a18e8f3b4526d16b82-e942e5928aab48eb9944179c292b8d1b on tablet e942e5928aab48eb9944179c292b8d1b with peer 6245203dd9bb46a18e8f3b4526d16b82
I20240503 07:58:36.502522 3949 raft_consensus.cc:1203] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:36.504150 4306 raft_consensus.cc:2932] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6245203dd9bb46a18e8f3b4526d16b82 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } } }
I20240503 07:58:36.504829 3949 raft_consensus.cc:2932] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6245203dd9bb46a18e8f3b4526d16b82 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } } }
I20240503 07:58:36.507709 2260 catalog_manager.cc:5484] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d reported cstate change: config changed from index -1 to 4, VOTER 6245203dd9bb46a18e8f3b4526d16b82 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "80435ecadfcf4ede849c7a97abea952d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } } }
I20240503 07:58:36.529225 2537 consensus_queue.cc:237] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } }
W20240503 07:58:36.534206 2470 consensus_peers.cc:485] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d -> Peer 27715b19d4ad40d89c2d12e66aca6a34 (127.0.99.135:34969): Couldn't send request to peer 27715b19d4ad40d89c2d12e66aca6a34. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ed9251d58be42d3841e8eb72b520353. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:37.091958 4330 ts_tablet_manager.cc:902] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34: Initiating tablet copy from peer 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895)
I20240503 07:58:37.092739 4330 tablet_copy_client.cc:308] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33895
I20240503 07:58:37.096563 2557 tablet_copy_service.cc:140] P 80435ecadfcf4ede849c7a97abea952d: Received BeginTabletCopySession request for tablet 6ed9251d58be42d3841e8eb72b520353 from peer 27715b19d4ad40d89c2d12e66aca6a34 ({username='slave'} at 127.0.99.135:46489)
I20240503 07:58:37.096678 2557 tablet_copy_service.cc:161] P 80435ecadfcf4ede849c7a97abea952d: Beginning new tablet copy session on tablet 6ed9251d58be42d3841e8eb72b520353 from peer 27715b19d4ad40d89c2d12e66aca6a34 at {username='slave'} at 127.0.99.135:46489: session id = 27715b19d4ad40d89c2d12e66aca6a34-6ed9251d58be42d3841e8eb72b520353
I20240503 07:58:37.097432 2557 tablet_copy_source_session.cc:205] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:37.100695 4330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed9251d58be42d3841e8eb72b520353. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:37.106276 4330 tablet_copy_client.cc:783] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:37.106511 4330 tablet_copy_client.cc:647] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:37.108115 4330 tablet_copy_client.cc:523] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:37.109866 4330 tablet_bootstrap.cc:492] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34: Bootstrap starting.
I20240503 07:58:37.115257 4330 log.cc:824] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:37.135653 4330 tablet_bootstrap.cc:492] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:58:37.136078 4330 tablet_bootstrap.cc:492] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34: Bootstrap complete.
I20240503 07:58:37.136417 4330 ts_tablet_manager.cc:1367] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34: Time spent bootstrapping tablet: real 0.027s user 0.026s sys 0.000s
I20240503 07:58:37.138602 4330 raft_consensus.cc:348] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:37.139218 4330 raft_consensus.cc:729] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27715b19d4ad40d89c2d12e66aca6a34, State: Initialized, Role: FOLLOWER
I20240503 07:58:37.139524 4330 consensus_queue.cc:260] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } }
I20240503 07:58:37.140340 4330 ts_tablet_manager.cc:1398] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 07:58:37.141173 2557 tablet_copy_service.cc:342] P 80435ecadfcf4ede849c7a97abea952d: Request end of tablet copy session 27715b19d4ad40d89c2d12e66aca6a34-6ed9251d58be42d3841e8eb72b520353 received from {username='slave'} at 127.0.99.135:46489
I20240503 07:58:37.141270 2557 tablet_copy_service.cc:433] P 80435ecadfcf4ede849c7a97abea952d: ending tablet copy session 27715b19d4ad40d89c2d12e66aca6a34-6ed9251d58be42d3841e8eb72b520353 on tablet 6ed9251d58be42d3841e8eb72b520353 with peer 27715b19d4ad40d89c2d12e66aca6a34
I20240503 07:58:37.536453 4080 raft_consensus.cc:1203] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:37.537714 4339 raft_consensus.cc:2932] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 27715b19d4ad40d89c2d12e66aca6a34 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } } }
I20240503 07:58:37.538365 4080 raft_consensus.cc:2932] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 27715b19d4ad40d89c2d12e66aca6a34 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } } }
I20240503 07:58:37.539264 4339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:42024 (request call id 0) took 1010 ms. Trace:
I20240503 07:58:37.539557 4339 rpcz_store.cc:276] 0503 07:58:36.528842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 07:58:36.528904 (+ 62us) service_pool.cc:225] Handling call
0503 07:58:36.529725 (+ 821us) log.cc:842] Serialized 257 byte log entry
0503 07:58:37.539243 (+1009518us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":53,"raft.queue_time_us":53,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"thread_start_us":94,"threads_started":1,"wal-append.queue_time_us":200,"wal-append.run_cpu_time_us":383,"wal-append.run_wall_time_us":1000464}
I20240503 07:58:37.541217 2264 catalog_manager.cc:5484] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 reported cstate change: config changed from index -1 to 4, VOTER 27715b19d4ad40d89c2d12e66aca6a34 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "80435ecadfcf4ede849c7a97abea952d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } } }
I20240503 07:58:37.560442 2537 consensus_queue.cc:237] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
W20240503 07:58:37.565188 2470 consensus_peers.cc:485] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d -> Peer e4a21231285448c88ef7465d265f20b5 (127.0.99.131:38167): Couldn't send request to peer e4a21231285448c88ef7465d265f20b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12be85cd12b84b4aa6389177679cd9d3. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:38.016052 4365 ts_tablet_manager.cc:902] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5: Initiating tablet copy from peer 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895)
I20240503 07:58:38.016898 4365 tablet_copy_client.cc:308] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33895
I20240503 07:58:38.020753 2557 tablet_copy_service.cc:140] P 80435ecadfcf4ede849c7a97abea952d: Received BeginTabletCopySession request for tablet 12be85cd12b84b4aa6389177679cd9d3 from peer e4a21231285448c88ef7465d265f20b5 ({username='slave'} at 127.0.99.131:60765)
I20240503 07:58:38.020879 2557 tablet_copy_service.cc:161] P 80435ecadfcf4ede849c7a97abea952d: Beginning new tablet copy session on tablet 12be85cd12b84b4aa6389177679cd9d3 from peer e4a21231285448c88ef7465d265f20b5 at {username='slave'} at 127.0.99.131:60765: session id = e4a21231285448c88ef7465d265f20b5-12be85cd12b84b4aa6389177679cd9d3
I20240503 07:58:38.021673 2557 tablet_copy_source_session.cc:205] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:38.023480 4365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12be85cd12b84b4aa6389177679cd9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:38.026278 4365 tablet_copy_client.cc:783] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:38.026495 4365 tablet_copy_client.cc:647] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:38.028054 4365 tablet_copy_client.cc:523] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:38.029362 4365 tablet_bootstrap.cc:492] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5: Bootstrap starting.
I20240503 07:58:38.049964 4365 tablet_bootstrap.cc:492] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:58:38.050405 4365 tablet_bootstrap.cc:492] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5: Bootstrap complete.
I20240503 07:58:38.050596 4365 ts_tablet_manager.cc:1367] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5: Time spent bootstrapping tablet: real 0.021s user 0.022s sys 0.000s
I20240503 07:58:38.051158 4365 raft_consensus.cc:348] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:38.051481 4365 raft_consensus.cc:729] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a21231285448c88ef7465d265f20b5, State: Initialized, Role: FOLLOWER
I20240503 07:58:38.051589 4365 consensus_queue.cc:260] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:38.051939 4365 ts_tablet_manager.cc:1398] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:38.052727 2557 tablet_copy_service.cc:342] P 80435ecadfcf4ede849c7a97abea952d: Request end of tablet copy session e4a21231285448c88ef7465d265f20b5-12be85cd12b84b4aa6389177679cd9d3 received from {username='slave'} at 127.0.99.131:60765
I20240503 07:58:38.052817 2557 tablet_copy_service.cc:433] P 80435ecadfcf4ede849c7a97abea952d: ending tablet copy session e4a21231285448c88ef7465d265f20b5-12be85cd12b84b4aa6389177679cd9d3 on tablet 12be85cd12b84b4aa6389177679cd9d3 with peer e4a21231285448c88ef7465d265f20b5
I20240503 07:58:38.611528 3550 raft_consensus.cc:1203] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:38.612913 4374 raft_consensus.cc:2932] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e4a21231285448c88ef7465d265f20b5 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } } }
I20240503 07:58:38.613780 3550 raft_consensus.cc:2932] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e4a21231285448c88ef7465d265f20b5 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } } }
I20240503 07:58:38.614459 4374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:42032 (request call id 0) took 1054 ms. Trace:
I20240503 07:58:38.614576 4374 rpcz_store.cc:276] 0503 07:58:37.560037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 07:58:37.560093 (+ 56us) service_pool.cc:225] Handling call
0503 07:58:37.560927 (+ 834us) log.cc:842] Serialized 257 byte log entry
0503 07:58:38.614442 (+1053515us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":49,"raft.queue_time_us":40,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":142,"thread_start_us":75,"threads_started":1,"wal-append.queue_time_us":141,"wal-append.run_cpu_time_us":327,"wal-append.run_wall_time_us":1000381}
I20240503 07:58:38.616307 2264 catalog_manager.cc:5484] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d reported cstate change: config changed from index -1 to 4, VOTER e4a21231285448c88ef7465d265f20b5 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "80435ecadfcf4ede849c7a97abea952d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } } }
I20240503 07:58:38.634698 2406 consensus_queue.cc:237] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } }
W20240503 07:58:38.636219 2340 consensus_peers.cc:485] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b -> Peer 1147cda1821744448cfea2164c8f3d58 (127.0.99.132:46551): Couldn't send request to peer 1147cda1821744448cfea2164c8f3d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9e176777a54493f94cdf609210874b6. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:39.171665 4399 ts_tablet_manager.cc:902] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58: Initiating tablet copy from peer 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:39.172169 4399 tablet_copy_client.cc:308] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35381
I20240503 07:58:39.172717 2426 tablet_copy_service.cc:140] P 9405f157903141179a86c810396a416b: Received BeginTabletCopySession request for tablet e9e176777a54493f94cdf609210874b6 from peer 1147cda1821744448cfea2164c8f3d58 ({username='slave'} at 127.0.99.132:51409)
I20240503 07:58:39.172835 2426 tablet_copy_service.cc:161] P 9405f157903141179a86c810396a416b: Beginning new tablet copy session on tablet e9e176777a54493f94cdf609210874b6 from peer 1147cda1821744448cfea2164c8f3d58 at {username='slave'} at 127.0.99.132:51409: session id = 1147cda1821744448cfea2164c8f3d58-e9e176777a54493f94cdf609210874b6
I20240503 07:58:39.173626 2426 tablet_copy_source_session.cc:205] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:39.174578 4399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e176777a54493f94cdf609210874b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:39.176470 4399 tablet_copy_client.cc:783] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:39.176633 4399 tablet_copy_client.cc:647] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:39.177899 4399 tablet_copy_client.cc:523] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:39.179124 4399 tablet_bootstrap.cc:492] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58: Bootstrap starting.
I20240503 07:58:39.197237 4399 tablet_bootstrap.cc:492] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:58:39.197593 4399 tablet_bootstrap.cc:492] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58: Bootstrap complete.
I20240503 07:58:39.197784 4399 ts_tablet_manager.cc:1367] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20240503 07:58:39.198418 4399 raft_consensus.cc:348] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:39.198668 4399 raft_consensus.cc:729] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1147cda1821744448cfea2164c8f3d58, State: Initialized, Role: FOLLOWER
I20240503 07:58:39.198786 4399 consensus_queue.cc:260] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } }
I20240503 07:58:39.199148 4399 ts_tablet_manager.cc:1398] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 07:58:39.199847 2426 tablet_copy_service.cc:342] P 9405f157903141179a86c810396a416b: Request end of tablet copy session 1147cda1821744448cfea2164c8f3d58-e9e176777a54493f94cdf609210874b6 received from {username='slave'} at 127.0.99.132:51409
I20240503 07:58:39.199944 2426 tablet_copy_service.cc:433] P 9405f157903141179a86c810396a416b: ending tablet copy session 1147cda1821744448cfea2164c8f3d58-e9e176777a54493f94cdf609210874b6 on tablet e9e176777a54493f94cdf609210874b6 with peer 1147cda1821744448cfea2164c8f3d58
I20240503 07:58:39.721671 3687 raft_consensus.cc:1203] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:39.722759 4407 raft_consensus.cc:2932] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1147cda1821744448cfea2164c8f3d58 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } } }
I20240503 07:58:39.723233 3687 raft_consensus.cc:2932] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1147cda1821744448cfea2164c8f3d58 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } } }
I20240503 07:58:39.724185 4407 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:50786 (request call id 0) took 1089 ms. Trace:
I20240503 07:58:39.724305 4407 rpcz_store.cc:276] 0503 07:58:38.634260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 07:58:38.634329 (+ 69us) service_pool.cc:225] Handling call
0503 07:58:38.635145 (+ 816us) log.cc:842] Serialized 257 byte log entry
0503 07:58:39.724167 (+1089022us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":55,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":161,"spinlock_wait_cycles":11648,"thread_start_us":87,"threads_started":1,"wal-append.queue_time_us":195,"wal-append.run_cpu_time_us":367,"wal-append.run_wall_time_us":1000438}
I20240503 07:58:39.725462 2260 catalog_manager.cc:5484] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 reported cstate change: config changed from index -1 to 4, VOTER 1147cda1821744448cfea2164c8f3d58 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } } }
I20240503 07:58:39.744740 2406 consensus_queue.cc:237] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } }
W20240503 07:58:39.746260 2342 consensus_peers.cc:485] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b -> Peer 6fa4f16dd2a545dbb01a0b260ce68744 (127.0.99.133:43115): Couldn't send request to peer 6fa4f16dd2a545dbb01a0b260ce68744. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527a672a8bd44251a56c5e47cc068e93. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:40.141834 4432 ts_tablet_manager.cc:902] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744: Initiating tablet copy from peer 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:40.142422 4432 tablet_copy_client.cc:308] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35381
I20240503 07:58:40.142964 2426 tablet_copy_service.cc:140] P 9405f157903141179a86c810396a416b: Received BeginTabletCopySession request for tablet 527a672a8bd44251a56c5e47cc068e93 from peer 6fa4f16dd2a545dbb01a0b260ce68744 ({username='slave'} at 127.0.99.133:43235)
I20240503 07:58:40.143082 2426 tablet_copy_service.cc:161] P 9405f157903141179a86c810396a416b: Beginning new tablet copy session on tablet 527a672a8bd44251a56c5e47cc068e93 from peer 6fa4f16dd2a545dbb01a0b260ce68744 at {username='slave'} at 127.0.99.133:43235: session id = 6fa4f16dd2a545dbb01a0b260ce68744-527a672a8bd44251a56c5e47cc068e93
I20240503 07:58:40.143913 2426 tablet_copy_source_session.cc:205] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:40.144814 4432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527a672a8bd44251a56c5e47cc068e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:40.146647 4432 tablet_copy_client.cc:783] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:40.146806 4432 tablet_copy_client.cc:647] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:40.147888 4432 tablet_copy_client.cc:523] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:40.149080 4432 tablet_bootstrap.cc:492] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744: Bootstrap starting.
I20240503 07:58:40.166469 4432 tablet_bootstrap.cc:492] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:58:40.166744 4432 tablet_bootstrap.cc:492] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744: Bootstrap complete.
I20240503 07:58:40.166914 4432 ts_tablet_manager.cc:1367] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.001s
I20240503 07:58:40.167464 4432 raft_consensus.cc:348] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:40.167696 4432 raft_consensus.cc:729] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa4f16dd2a545dbb01a0b260ce68744, State: Initialized, Role: FOLLOWER
I20240503 07:58:40.167816 4432 consensus_queue.cc:260] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } }
I20240503 07:58:40.168329 4432 ts_tablet_manager.cc:1398] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:40.169005 2426 tablet_copy_service.cc:342] P 9405f157903141179a86c810396a416b: Request end of tablet copy session 6fa4f16dd2a545dbb01a0b260ce68744-527a672a8bd44251a56c5e47cc068e93 received from {username='slave'} at 127.0.99.133:43235
I20240503 07:58:40.169102 2426 tablet_copy_service.cc:433] P 9405f157903141179a86c810396a416b: ending tablet copy session 6fa4f16dd2a545dbb01a0b260ce68744-527a672a8bd44251a56c5e47cc068e93 on tablet 527a672a8bd44251a56c5e47cc068e93 with peer 6fa4f16dd2a545dbb01a0b260ce68744
I20240503 07:58:40.705855 3818 raft_consensus.cc:1203] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:40.706921 4440 raft_consensus.cc:2932] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6fa4f16dd2a545dbb01a0b260ce68744 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } } }
I20240503 07:58:40.707357 3818 raft_consensus.cc:2932] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6fa4f16dd2a545dbb01a0b260ce68744 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } } }
I20240503 07:58:40.710105 2260 catalog_manager.cc:5484] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b reported cstate change: config changed from index -1 to 4, VOTER 6fa4f16dd2a545dbb01a0b260ce68744 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } } }
I20240503 07:58:40.728264 2537 consensus_queue.cc:237] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } }
W20240503 07:58:40.729826 2473 consensus_peers.cc:485] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d -> Peer 6245203dd9bb46a18e8f3b4526d16b82 (127.0.99.134:46183): Couldn't send request to peer 6245203dd9bb46a18e8f3b4526d16b82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52c02ae315844d90aa9454b9c022f27d. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:41.183248 4463 ts_tablet_manager.cc:902] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82: Initiating tablet copy from peer 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895)
I20240503 07:58:41.183723 4463 tablet_copy_client.cc:308] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33895
I20240503 07:58:41.184312 2557 tablet_copy_service.cc:140] P 80435ecadfcf4ede849c7a97abea952d: Received BeginTabletCopySession request for tablet 52c02ae315844d90aa9454b9c022f27d from peer 6245203dd9bb46a18e8f3b4526d16b82 ({username='slave'} at 127.0.99.134:43697)
I20240503 07:58:41.184444 2557 tablet_copy_service.cc:161] P 80435ecadfcf4ede849c7a97abea952d: Beginning new tablet copy session on tablet 52c02ae315844d90aa9454b9c022f27d from peer 6245203dd9bb46a18e8f3b4526d16b82 at {username='slave'} at 127.0.99.134:43697: session id = 6245203dd9bb46a18e8f3b4526d16b82-52c02ae315844d90aa9454b9c022f27d
I20240503 07:58:41.185300 2557 tablet_copy_source_session.cc:205] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:41.186611 4463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c02ae315844d90aa9454b9c022f27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:41.188741 4463 tablet_copy_client.cc:783] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:41.188901 4463 tablet_copy_client.cc:647] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:41.190184 4463 tablet_copy_client.cc:523] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:41.191414 4463 tablet_bootstrap.cc:492] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82: Bootstrap starting.
I20240503 07:58:41.208015 4463 tablet_bootstrap.cc:492] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:58:41.208377 4463 tablet_bootstrap.cc:492] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82: Bootstrap complete.
I20240503 07:58:41.208571 4463 ts_tablet_manager.cc:1367] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.002s
I20240503 07:58:41.209172 4463 raft_consensus.cc:348] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:41.209435 4463 raft_consensus.cc:729] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6245203dd9bb46a18e8f3b4526d16b82, State: Initialized, Role: FOLLOWER
I20240503 07:58:41.209544 4463 consensus_queue.cc:260] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } }
I20240503 07:58:41.209903 4463 ts_tablet_manager.cc:1398] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:58:41.210629 2557 tablet_copy_service.cc:342] P 80435ecadfcf4ede849c7a97abea952d: Request end of tablet copy session 6245203dd9bb46a18e8f3b4526d16b82-52c02ae315844d90aa9454b9c022f27d received from {username='slave'} at 127.0.99.134:43697
I20240503 07:58:41.210721 2557 tablet_copy_service.cc:433] P 80435ecadfcf4ede849c7a97abea952d: ending tablet copy session 6245203dd9bb46a18e8f3b4526d16b82-52c02ae315844d90aa9454b9c022f27d on tablet 52c02ae315844d90aa9454b9c022f27d with peer 6245203dd9bb46a18e8f3b4526d16b82
I20240503 07:58:41.693884 3949 raft_consensus.cc:1203] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:41.694833 4470 raft_consensus.cc:2932] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6245203dd9bb46a18e8f3b4526d16b82 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } } }
I20240503 07:58:41.695264 3949 raft_consensus.cc:2932] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6245203dd9bb46a18e8f3b4526d16b82 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } } }
I20240503 07:58:41.698055 2260 catalog_manager.cc:5484] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d reported cstate change: config changed from index -1 to 4, VOTER 6245203dd9bb46a18e8f3b4526d16b82 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "80435ecadfcf4ede849c7a97abea952d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } } }
I20240503 07:58:41.721048 2406 consensus_queue.cc:237] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } }
W20240503 07:58:41.725553 2339 consensus_peers.cc:485] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b -> Peer 27715b19d4ad40d89c2d12e66aca6a34 (127.0.99.135:34969): Couldn't send request to peer 27715b19d4ad40d89c2d12e66aca6a34. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1246a3d482304f02bbfe1c97c2e5a1ad. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:42.168948 4495 ts_tablet_manager.cc:902] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34: Initiating tablet copy from peer 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:42.169482 4495 tablet_copy_client.cc:308] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35381
I20240503 07:58:42.173214 2426 tablet_copy_service.cc:140] P 9405f157903141179a86c810396a416b: Received BeginTabletCopySession request for tablet 1246a3d482304f02bbfe1c97c2e5a1ad from peer 27715b19d4ad40d89c2d12e66aca6a34 ({username='slave'} at 127.0.99.135:41557)
I20240503 07:58:42.173339 2426 tablet_copy_service.cc:161] P 9405f157903141179a86c810396a416b: Beginning new tablet copy session on tablet 1246a3d482304f02bbfe1c97c2e5a1ad from peer 27715b19d4ad40d89c2d12e66aca6a34 at {username='slave'} at 127.0.99.135:41557: session id = 27715b19d4ad40d89c2d12e66aca6a34-1246a3d482304f02bbfe1c97c2e5a1ad
I20240503 07:58:42.174098 2426 tablet_copy_source_session.cc:205] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:42.175171 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1246a3d482304f02bbfe1c97c2e5a1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:42.177081 4495 tablet_copy_client.cc:783] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:42.177227 4495 tablet_copy_client.cc:647] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:42.178445 4495 tablet_copy_client.cc:523] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:42.179798 4495 tablet_bootstrap.cc:492] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34: Bootstrap starting.
I20240503 07:58:42.197903 4495 tablet_bootstrap.cc:492] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:58:42.198228 4495 tablet_bootstrap.cc:492] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34: Bootstrap complete.
I20240503 07:58:42.198429 4495 ts_tablet_manager.cc:1367] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.009s
I20240503 07:58:42.198982 4495 raft_consensus.cc:348] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:42.199210 4495 raft_consensus.cc:729] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27715b19d4ad40d89c2d12e66aca6a34, State: Initialized, Role: FOLLOWER
I20240503 07:58:42.199316 4495 consensus_queue.cc:260] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } }
I20240503 07:58:42.199669 4495 ts_tablet_manager.cc:1398] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:42.200330 2426 tablet_copy_service.cc:342] P 9405f157903141179a86c810396a416b: Request end of tablet copy session 27715b19d4ad40d89c2d12e66aca6a34-1246a3d482304f02bbfe1c97c2e5a1ad received from {username='slave'} at 127.0.99.135:41557
I20240503 07:58:42.200414 2426 tablet_copy_service.cc:433] P 9405f157903141179a86c810396a416b: ending tablet copy session 27715b19d4ad40d89c2d12e66aca6a34-1246a3d482304f02bbfe1c97c2e5a1ad on tablet 1246a3d482304f02bbfe1c97c2e5a1ad with peer 27715b19d4ad40d89c2d12e66aca6a34
I20240503 07:58:42.738600 4080 raft_consensus.cc:1203] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:42.739609 4504 raft_consensus.cc:2932] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 27715b19d4ad40d89c2d12e66aca6a34 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } } }
I20240503 07:58:42.740057 4080 raft_consensus.cc:2932] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 27715b19d4ad40d89c2d12e66aca6a34 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } } }
I20240503 07:58:42.741079 4504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:50816 (request call id 0) took 1020 ms. Trace:
I20240503 07:58:42.741176 4504 rpcz_store.cc:276] 0503 07:58:41.720675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 07:58:41.720746 (+ 71us) service_pool.cc:225] Handling call
0503 07:58:41.721524 (+ 778us) log.cc:842] Serialized 257 byte log entry
0503 07:58:42.741064 (+1019540us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":44,"raft.queue_time_us":46,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":82,"threads_started":1,"wal-append.queue_time_us":161,"wal-append.run_cpu_time_us":320,"wal-append.run_wall_time_us":1000399}
I20240503 07:58:42.742306 2256 catalog_manager.cc:5484] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 reported cstate change: config changed from index -1 to 4, VOTER 27715b19d4ad40d89c2d12e66aca6a34 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } } }
I20240503 07:58:42.800426 3667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:42.808683 2385 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:58:42.809746 3929 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:42.810832 3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:42.811398 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:42.822646 2517 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:58:42.840261 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:42.864180 2537 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52c02ae315844d90aa9454b9c022f27d"
dest_uuid: "80435ecadfcf4ede849c7a97abea952d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42084
I20240503 07:58:42.864375 2537 raft_consensus.cc:595] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:42.919473 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:42.919745 3667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:42.924822 3929 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:42.934952 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:42.934989 2385 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:58:42.941355 3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:42.941524 2517 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:58:43.031172 3667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.033221 3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.036100 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.036612 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.041949 2385 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:58:43.045341 2517 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:58:43.061455 3929 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.087226 2406 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "527a672a8bd44251a56c5e47cc068e93"
dest_uuid: "9405f157903141179a86c810396a416b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50860
I20240503 07:58:43.087427 2406 raft_consensus.cc:595] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:43.139509 3667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.139772 3929 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.140478 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.144268 2517 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:58:43.153764 3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.155894 2385 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:58:43.167068 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.245896 3667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.249393 3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.255722 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.260574 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.266398 4407 raft_consensus.cc:979] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b: : Instructing follower 6fa4f16dd2a545dbb01a0b260ce68744 to start an election
I20240503 07:58:43.266513 4407 raft_consensus.cc:1067] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Signalling peer 6fa4f16dd2a545dbb01a0b260ce68744 to start an election
I20240503 07:58:43.267293 3818 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "527a672a8bd44251a56c5e47cc068e93"
dest_uuid: "6fa4f16dd2a545dbb01a0b260ce68744"
from {username='slave'} at 127.0.99.129:43569
I20240503 07:58:43.267517 3818 raft_consensus.cc:482] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:43.267607 3929 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.267628 3818 raft_consensus.cc:3037] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:43.268292 2517 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:58:43.268805 3818 raft_consensus.cc:504] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } }
I20240503 07:58:43.269922 3818 leader_election.cc:288] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [CANDIDATE]: Term 2 election: Requested vote from peers 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:43.274555 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527a672a8bd44251a56c5e47cc068e93" candidate_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9405f157903141179a86c810396a416b"
I20240503 07:58:43.274776 2406 raft_consensus.cc:3032] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:43.274823 2406 raft_consensus.cc:729] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:43.274998 2406 consensus_queue.cc:260] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } }
I20240503 07:58:43.275264 2406 raft_consensus.cc:3037] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:43.276360 2406 raft_consensus.cc:2454] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa4f16dd2a545dbb01a0b260ce68744 in term 2.
I20240503 07:58:43.276890 3751 leader_election.cc:302] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6fa4f16dd2a545dbb01a0b260ce68744, 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:43.277307 4705 raft_consensus.cc:2781] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:43.277545 4705 raft_consensus.cc:686] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 2 LEADER]: Becoming Leader. State: Replica: 6fa4f16dd2a545dbb01a0b260ce68744, State: Running, Role: LEADER
I20240503 07:58:43.277925 4705 consensus_queue.cc:237] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } }
I20240503 07:58:43.280557 2260 catalog_manager.cc:5484] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 reported cstate change: term changed from 1 to 2, leader changed from 9405f157903141179a86c810396a416b (127.0.99.129) to 6fa4f16dd2a545dbb01a0b260ce68744 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } } }
I20240503 07:58:43.289187 2385 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:58:43.310794 2406 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5031a2c9d5af4289928e0e43488931f8"
dest_uuid: "9405f157903141179a86c810396a416b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50888
I20240503 07:58:43.310958 2406 raft_consensus.cc:595] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:43.322190 4407 raft_consensus.cc:979] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b: : Instructing follower 6fa4f16dd2a545dbb01a0b260ce68744 to start an election
I20240503 07:58:43.322320 4407 raft_consensus.cc:1067] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Signalling peer 6fa4f16dd2a545dbb01a0b260ce68744 to start an election
I20240503 07:58:43.322798 3818 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5031a2c9d5af4289928e0e43488931f8"
dest_uuid: "6fa4f16dd2a545dbb01a0b260ce68744"
from {username='slave'} at 127.0.99.129:43569
I20240503 07:58:43.322950 3818 raft_consensus.cc:482] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:43.323009 3818 raft_consensus.cc:3037] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:43.324077 3818 raft_consensus.cc:504] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } }
I20240503 07:58:43.324419 3818 leader_election.cc:288] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [CANDIDATE]: Term 2 election: Requested vote from peers 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:43.324808 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5031a2c9d5af4289928e0e43488931f8" candidate_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9405f157903141179a86c810396a416b"
I20240503 07:58:43.324934 2406 raft_consensus.cc:3032] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:43.324977 2406 raft_consensus.cc:729] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:43.325131 2406 consensus_queue.cc:260] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } }
I20240503 07:58:43.325271 2406 raft_consensus.cc:3037] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:43.326260 2406 raft_consensus.cc:2454] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa4f16dd2a545dbb01a0b260ce68744 in term 2.
I20240503 07:58:43.326589 3751 leader_election.cc:302] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6fa4f16dd2a545dbb01a0b260ce68744, 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:43.326851 4705 raft_consensus.cc:2781] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:43.326967 4705 raft_consensus.cc:686] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 2 LEADER]: Becoming Leader. State: Replica: 6fa4f16dd2a545dbb01a0b260ce68744, State: Running, Role: LEADER
I20240503 07:58:43.327121 4705 consensus_queue.cc:237] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } }
I20240503 07:58:43.328624 2260 catalog_manager.cc:5484] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 reported cstate change: term changed from 1 to 2, leader changed from 9405f157903141179a86c810396a416b (127.0.99.129) to 6fa4f16dd2a545dbb01a0b260ce68744 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } } }
I20240503 07:58:43.340991 4374 raft_consensus.cc:979] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d: : Instructing follower 6245203dd9bb46a18e8f3b4526d16b82 to start an election
I20240503 07:58:43.341130 4374 raft_consensus.cc:1067] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Signalling peer 6245203dd9bb46a18e8f3b4526d16b82 to start an election
I20240503 07:58:43.341774 3949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52c02ae315844d90aa9454b9c022f27d"
dest_uuid: "6245203dd9bb46a18e8f3b4526d16b82"
from {username='slave'} at 127.0.99.130:44409
I20240503 07:58:43.342015 3949 raft_consensus.cc:482] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:43.342128 3949 raft_consensus.cc:3037] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:43.343353 3949 raft_consensus.cc:504] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } }
I20240503 07:58:43.344449 3949 leader_election.cc:288] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [CANDIDATE]: Term 2 election: Requested vote from peers 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895)
I20240503 07:58:43.344921 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c02ae315844d90aa9454b9c022f27d" candidate_uuid: "6245203dd9bb46a18e8f3b4526d16b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "80435ecadfcf4ede849c7a97abea952d"
I20240503 07:58:43.345158 2537 raft_consensus.cc:3032] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:43.345252 2537 raft_consensus.cc:729] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Running, Role: LEADER
I20240503 07:58:43.345434 2537 consensus_queue.cc:260] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } }
I20240503 07:58:43.345711 2537 raft_consensus.cc:3037] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:43.346810 2537 raft_consensus.cc:2454] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6245203dd9bb46a18e8f3b4526d16b82 in term 2.
I20240503 07:58:43.347242 3885 leader_election.cc:302] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6245203dd9bb46a18e8f3b4526d16b82, 80435ecadfcf4ede849c7a97abea952d; no voters:
I20240503 07:58:43.347606 4734 raft_consensus.cc:2781] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:43.347781 4734 raft_consensus.cc:686] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [term 2 LEADER]: Becoming Leader. State: Replica: 6245203dd9bb46a18e8f3b4526d16b82, State: Running, Role: LEADER
I20240503 07:58:43.348079 4734 consensus_queue.cc:237] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } }
I20240503 07:58:43.350576 2260 catalog_manager.cc:5484] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 reported cstate change: term changed from 1 to 2, leader changed from 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) to 6245203dd9bb46a18e8f3b4526d16b82 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "6245203dd9bb46a18e8f3b4526d16b82" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } } }
I20240503 07:58:43.376070 3667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.376346 3929 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.377043 2517 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:43.379961 2385 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:43.382185 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.384990 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.391130 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:43.486862 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.487743 3929 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.491343 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:43.492827 2385 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:43.498929 3667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.499594 2517 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:43.522964 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.544901 2406 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dea872e4b037445c93067135349f57f4"
dest_uuid: "9405f157903141179a86c810396a416b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50926
I20240503 07:58:43.545076 2406 raft_consensus.cc:595] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:43.596932 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.600232 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:43.601295 3667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.603233 2517 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:43.605276 3929 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.612037 2385 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:43.628796 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.714265 3667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.717602 2385 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:43.718541 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:43.728415 3929 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.731493 2517 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:43.732152 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.740938 2405 raft_consensus.cc:1261] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Refusing update from remote peer 6fa4f16dd2a545dbb01a0b260ce68744: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:58:43.740938 2406 raft_consensus.cc:1261] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Refusing update from remote peer 6fa4f16dd2a545dbb01a0b260ce68744: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:58:43.741717 4706 consensus_queue.cc:1035] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:43.742859 4706 consensus_queue.cc:1035] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:43.752375 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.778008 2537 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e942e5928aab48eb9944179c292b8d1b"
dest_uuid: "80435ecadfcf4ede849c7a97abea952d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42168
I20240503 07:58:43.778172 2537 raft_consensus.cc:595] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:43.834077 3667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.835041 3929 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.837427 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.837713 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:43.844079 2537 raft_consensus.cc:1261] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Refusing update from remote peer 6245203dd9bb46a18e8f3b4526d16b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:58:43.845340 4734 consensus_queue.cc:1035] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:43.862569 2385 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:43.864420 2517 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:43.867424 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.891201 4407 raft_consensus.cc:979] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b: : Instructing follower 1147cda1821744448cfea2164c8f3d58 to start an election
I20240503 07:58:43.891346 4407 raft_consensus.cc:1067] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Signalling peer 1147cda1821744448cfea2164c8f3d58 to start an election
I20240503 07:58:43.891901 3687 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dea872e4b037445c93067135349f57f4"
dest_uuid: "1147cda1821744448cfea2164c8f3d58"
from {username='slave'} at 127.0.99.129:49929
I20240503 07:58:43.892134 3687 raft_consensus.cc:482] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:43.892261 3687 raft_consensus.cc:3037] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:43.893324 3687 raft_consensus.cc:504] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } }
I20240503 07:58:43.894479 3687 leader_election.cc:288] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [CANDIDATE]: Term 2 election: Requested vote from peers 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:43.894853 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea872e4b037445c93067135349f57f4" candidate_uuid: "1147cda1821744448cfea2164c8f3d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9405f157903141179a86c810396a416b"
I20240503 07:58:43.895001 2406 raft_consensus.cc:3032] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:43.895046 2406 raft_consensus.cc:729] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:43.895200 2406 consensus_queue.cc:260] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } }
I20240503 07:58:43.895331 2406 raft_consensus.cc:3037] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:43.896111 2406 raft_consensus.cc:2454] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1147cda1821744448cfea2164c8f3d58 in term 2.
I20240503 07:58:43.896545 3620 leader_election.cc:302] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1147cda1821744448cfea2164c8f3d58, 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:43.897032 4917 raft_consensus.cc:2781] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:43.897212 4917 raft_consensus.cc:686] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [term 2 LEADER]: Becoming Leader. State: Replica: 1147cda1821744448cfea2164c8f3d58, State: Running, Role: LEADER
I20240503 07:58:43.897576 4917 consensus_queue.cc:237] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } }
I20240503 07:58:43.900221 2264 catalog_manager.cc:5484] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 reported cstate change: term changed from 1 to 2, leader changed from 9405f157903141179a86c810396a416b (127.0.99.129) to 1147cda1821744448cfea2164c8f3d58 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "1147cda1821744448cfea2164c8f3d58" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } } }
I20240503 07:58:43.956205 3929 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.957698 3667 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:43.962400 2385 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:58:43.962675 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:43.970741 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:43.976895 2517 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:43.978819 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.013628 2406 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1246a3d482304f02bbfe1c97c2e5a1ad"
dest_uuid: "9405f157903141179a86c810396a416b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50990
I20240503 07:58:44.013789 2406 raft_consensus.cc:595] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:44.044368 4374 raft_consensus.cc:979] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d: : Instructing follower 6245203dd9bb46a18e8f3b4526d16b82 to start an election
I20240503 07:58:44.044485 4374 raft_consensus.cc:1067] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Signalling peer 6245203dd9bb46a18e8f3b4526d16b82 to start an election
I20240503 07:58:44.044986 3949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e942e5928aab48eb9944179c292b8d1b"
dest_uuid: "6245203dd9bb46a18e8f3b4526d16b82"
from {username='slave'} at 127.0.99.130:44409
I20240503 07:58:44.045150 3949 raft_consensus.cc:482] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:44.045221 3949 raft_consensus.cc:3037] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:44.046195 3949 raft_consensus.cc:504] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } }
I20240503 07:58:44.046568 3949 leader_election.cc:288] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [CANDIDATE]: Term 2 election: Requested vote from peers 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895)
I20240503 07:58:44.046998 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e942e5928aab48eb9944179c292b8d1b" candidate_uuid: "6245203dd9bb46a18e8f3b4526d16b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "80435ecadfcf4ede849c7a97abea952d"
I20240503 07:58:44.047140 2537 raft_consensus.cc:3032] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:44.047184 2537 raft_consensus.cc:729] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Running, Role: LEADER
I20240503 07:58:44.047325 2537 consensus_queue.cc:260] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } }
I20240503 07:58:44.047487 2537 raft_consensus.cc:3037] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:44.048288 2537 raft_consensus.cc:2454] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6245203dd9bb46a18e8f3b4526d16b82 in term 2.
I20240503 07:58:44.048578 3885 leader_election.cc:302] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6245203dd9bb46a18e8f3b4526d16b82, 80435ecadfcf4ede849c7a97abea952d; no voters:
I20240503 07:58:44.048782 4908 raft_consensus.cc:2781] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:44.048902 4908 raft_consensus.cc:686] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [term 2 LEADER]: Becoming Leader. State: Replica: 6245203dd9bb46a18e8f3b4526d16b82, State: Running, Role: LEADER
I20240503 07:58:44.049046 4908 consensus_queue.cc:237] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } }
I20240503 07:58:44.050908 2257 catalog_manager.cc:5484] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 reported cstate change: term changed from 1 to 2, leader changed from 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) to 6245203dd9bb46a18e8f3b4526d16b82 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "6245203dd9bb46a18e8f3b4526d16b82" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } } }
I20240503 07:58:44.070253 3667 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.072146 4060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:44.076264 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.077451 2517 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:58:44.084270 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.084302 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.096320 2385 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:58:44.152833 4407 raft_consensus.cc:979] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b: : Instructing follower 27715b19d4ad40d89c2d12e66aca6a34 to start an election
I20240503 07:58:44.152953 4407 raft_consensus.cc:1067] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 1 LEADER]: Signalling peer 27715b19d4ad40d89c2d12e66aca6a34 to start an election
I20240503 07:58:44.153463 4080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1246a3d482304f02bbfe1c97c2e5a1ad"
dest_uuid: "27715b19d4ad40d89c2d12e66aca6a34"
from {username='slave'} at 127.0.99.129:58831
I20240503 07:58:44.153664 4080 raft_consensus.cc:482] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:44.153784 4080 raft_consensus.cc:3037] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:44.154974 4080 raft_consensus.cc:504] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } }
I20240503 07:58:44.155974 4080 leader_election.cc:288] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [CANDIDATE]: Term 2 election: Requested vote from peers 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:44.156335 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1246a3d482304f02bbfe1c97c2e5a1ad" candidate_uuid: "27715b19d4ad40d89c2d12e66aca6a34" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9405f157903141179a86c810396a416b"
I20240503 07:58:44.156484 2406 raft_consensus.cc:3032] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:44.156534 2406 raft_consensus.cc:729] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:44.156683 2406 consensus_queue.cc:260] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } }
I20240503 07:58:44.156857 2406 raft_consensus.cc:3037] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:44.157806 2406 raft_consensus.cc:2454] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27715b19d4ad40d89c2d12e66aca6a34 in term 2.
I20240503 07:58:44.158179 4013 leader_election.cc:302] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 27715b19d4ad40d89c2d12e66aca6a34, 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:44.158712 5010 raft_consensus.cc:2781] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:44.158905 5010 raft_consensus.cc:686] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [term 2 LEADER]: Becoming Leader. State: Replica: 27715b19d4ad40d89c2d12e66aca6a34, State: Running, Role: LEADER
I20240503 07:58:44.159271 5010 consensus_queue.cc:237] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } }
I20240503 07:58:44.161983 2260 catalog_manager.cc:5484] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 reported cstate change: term changed from 1 to 2, leader changed from 9405f157903141179a86c810396a416b (127.0.99.129) to 27715b19d4ad40d89c2d12e66aca6a34 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "27715b19d4ad40d89c2d12e66aca6a34" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } } }
I20240503 07:58:44.184501 3667 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.194972 2385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:44.199101 2517 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:58:44.201136 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.201372 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.202172 4060 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.202841 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.245501 2406 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9e176777a54493f94cdf609210874b6"
dest_uuid: "9405f157903141179a86c810396a416b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51026
I20240503 07:58:44.245667 2406 raft_consensus.cc:595] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:44.300539 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.302667 3667 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.303673 4060 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.304637 2385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:44.304711 2517 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:58:44.313040 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.321043 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.334416 2406 raft_consensus.cc:1261] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Refusing update from remote peer 1147cda1821744448cfea2164c8f3d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:58:44.335073 4917 consensus_queue.cc:1035] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:44.422489 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.422778 2517 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:58:44.423720 4060 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.424333 3667 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.425022 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.438525 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.438977 2385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:44.480768 2537 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ed9251d58be42d3841e8eb72b520353"
dest_uuid: "80435ecadfcf4ede849c7a97abea952d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42256
I20240503 07:58:44.480947 2537 raft_consensus.cc:595] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:44.536109 3667 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.538834 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.540436 4060 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.543000 2385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:44.545307 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.552352 2517 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:58:44.552352 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.595513 2406 raft_consensus.cc:1261] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Refusing update from remote peer 27715b19d4ad40d89c2d12e66aca6a34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:58:44.596052 5010 consensus_queue.cc:1035] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:44.647151 3667 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.650916 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.652849 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.653158 2517 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:58:44.660148 2537 raft_consensus.cc:1261] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Refusing update from remote peer 6245203dd9bb46a18e8f3b4526d16b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:58:44.660679 4908 consensus_queue.cc:1035] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:44.663466 4060 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.665060 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.665757 2385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:44.706826 2537 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "12be85cd12b84b4aa6389177679cd9d3"
dest_uuid: "80435ecadfcf4ede849c7a97abea952d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42274
I20240503 07:58:44.706995 2537 raft_consensus.cc:595] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:44.764164 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.764158 3667 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.765142 4060 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.770609 2517 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:58:44.770704 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:44.783736 2385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:44.785926 3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:44.801549 4407 raft_consensus.cc:979] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b: : Instructing follower 1147cda1821744448cfea2164c8f3d58 to start an election
I20240503 07:58:44.801669 4407 raft_consensus.cc:1067] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Signalling peer 1147cda1821744448cfea2164c8f3d58 to start an election
I20240503 07:58:44.802173 3687 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9e176777a54493f94cdf609210874b6"
dest_uuid: "1147cda1821744448cfea2164c8f3d58"
from {username='slave'} at 127.0.99.129:49929
I20240503 07:58:44.802335 3687 raft_consensus.cc:482] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:44.802423 3687 raft_consensus.cc:3037] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:44.803552 3687 raft_consensus.cc:504] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } }
I20240503 07:58:44.803910 3687 leader_election.cc:288] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [CANDIDATE]: Term 2 election: Requested vote from peers 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:44.804309 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e176777a54493f94cdf609210874b6" candidate_uuid: "1147cda1821744448cfea2164c8f3d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9405f157903141179a86c810396a416b"
I20240503 07:58:44.804461 2406 raft_consensus.cc:3032] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:44.804507 2406 raft_consensus.cc:729] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:44.804656 2406 consensus_queue.cc:260] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } }
I20240503 07:58:44.804752 4976 raft_consensus.cc:979] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d: : Instructing follower 27715b19d4ad40d89c2d12e66aca6a34 to start an election
I20240503 07:58:44.804811 2406 raft_consensus.cc:3037] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:44.804862 4976 raft_consensus.cc:1067] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Signalling peer 27715b19d4ad40d89c2d12e66aca6a34 to start an election
I20240503 07:58:44.805413 4080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ed9251d58be42d3841e8eb72b520353"
dest_uuid: "27715b19d4ad40d89c2d12e66aca6a34"
from {username='slave'} at 127.0.99.130:37719
I20240503 07:58:44.805579 4080 raft_consensus.cc:482] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:44.805630 4080 raft_consensus.cc:3037] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:44.805923 2406 raft_consensus.cc:2454] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1147cda1821744448cfea2164c8f3d58 in term 2.
I20240503 07:58:44.806272 3620 leader_election.cc:302] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1147cda1821744448cfea2164c8f3d58, 9405f157903141179a86c810396a416b; no voters:
I20240503 07:58:44.806552 4918 raft_consensus.cc:2781] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:44.806631 4918 raft_consensus.cc:686] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [term 2 LEADER]: Becoming Leader. State: Replica: 1147cda1821744448cfea2164c8f3d58, State: Running, Role: LEADER
I20240503 07:58:44.806650 4080 raft_consensus.cc:504] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } }
I20240503 07:58:44.806754 4918 consensus_queue.cc:237] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } }
I20240503 07:58:44.807076 4080 leader_election.cc:288] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [CANDIDATE]: Term 2 election: Requested vote from peers 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895)
I20240503 07:58:44.807585 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed9251d58be42d3841e8eb72b520353" candidate_uuid: "27715b19d4ad40d89c2d12e66aca6a34" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "80435ecadfcf4ede849c7a97abea952d"
I20240503 07:58:44.807708 2537 raft_consensus.cc:3032] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:44.807744 2537 raft_consensus.cc:729] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Running, Role: LEADER
I20240503 07:58:44.807893 2537 consensus_queue.cc:260] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } }
I20240503 07:58:44.808033 2537 raft_consensus.cc:3037] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:44.808558 2257 catalog_manager.cc:5484] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 reported cstate change: term changed from 1 to 2, leader changed from 9405f157903141179a86c810396a416b (127.0.99.129) to 1147cda1821744448cfea2164c8f3d58 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "1147cda1821744448cfea2164c8f3d58" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } } }
I20240503 07:58:44.809036 2537 raft_consensus.cc:2454] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27715b19d4ad40d89c2d12e66aca6a34 in term 2.
I20240503 07:58:44.809629 4016 leader_election.cc:302] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 27715b19d4ad40d89c2d12e66aca6a34, 80435ecadfcf4ede849c7a97abea952d; no voters:
I20240503 07:58:44.809841 5011 raft_consensus.cc:2781] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:44.809940 5011 raft_consensus.cc:686] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [term 2 LEADER]: Becoming Leader. State: Replica: 27715b19d4ad40d89c2d12e66aca6a34, State: Running, Role: LEADER
I20240503 07:58:44.810070 5011 consensus_queue.cc:237] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } }
I20240503 07:58:44.812101 2259 catalog_manager.cc:5484] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 reported cstate change: term changed from 1 to 2, leader changed from 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) to 27715b19d4ad40d89c2d12e66aca6a34 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "27715b19d4ad40d89c2d12e66aca6a34" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } } }
I20240503 07:58:44.945627 4976 raft_consensus.cc:979] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d: : Instructing follower e4a21231285448c88ef7465d265f20b5 to start an election
I20240503 07:58:44.945739 4976 raft_consensus.cc:1067] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Signalling peer e4a21231285448c88ef7465d265f20b5 to start an election
I20240503 07:58:44.946224 3550 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "12be85cd12b84b4aa6389177679cd9d3"
dest_uuid: "e4a21231285448c88ef7465d265f20b5"
from {username='slave'} at 127.0.99.130:49163
I20240503 07:58:44.946456 3550 raft_consensus.cc:482] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:44.946512 3550 raft_consensus.cc:3037] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:44.947324 3550 raft_consensus.cc:504] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:44.948071 3550 leader_election.cc:288] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [CANDIDATE]: Term 2 election: Requested vote from peers 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895)
I20240503 07:58:44.948488 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12be85cd12b84b4aa6389177679cd9d3" candidate_uuid: "e4a21231285448c88ef7465d265f20b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "80435ecadfcf4ede849c7a97abea952d"
I20240503 07:58:44.948616 2537 raft_consensus.cc:3032] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:44.948662 2537 raft_consensus.cc:729] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Running, Role: LEADER
I20240503 07:58:44.948832 2537 consensus_queue.cc:260] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:44.948972 2537 raft_consensus.cc:3037] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:44.949795 2537 raft_consensus.cc:2454] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4a21231285448c88ef7465d265f20b5 in term 2.
I20240503 07:58:44.950170 3485 leader_election.cc:302] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 80435ecadfcf4ede849c7a97abea952d, e4a21231285448c88ef7465d265f20b5; no voters:
I20240503 07:58:44.950517 5237 raft_consensus.cc:2781] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:44.950677 5237 raft_consensus.cc:686] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [term 2 LEADER]: Becoming Leader. State: Replica: e4a21231285448c88ef7465d265f20b5, State: Running, Role: LEADER
I20240503 07:58:44.950810 5237 consensus_queue.cc:237] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:44.952556 2259 catalog_manager.cc:5484] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 reported cstate change: term changed from 1 to 2, leader changed from 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) to e4a21231285448c88ef7465d265f20b5 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "e4a21231285448c88ef7465d265f20b5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } } }
I20240503 07:58:45.092525 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.093719 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.093720 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.094868 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.095955 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.095964 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.098704 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.159855 3949 consensus_queue.cc:237] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } }
I20240503 07:58:45.160787 5189 raft_consensus.cc:2932] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } } }
I20240503 07:58:45.163276 2259 catalog_manager.cc:5484] T 52c02ae315844d90aa9454b9c022f27d P 6245203dd9bb46a18e8f3b4526d16b82 reported cstate change: config changed from index 4 to 6, VOTER 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "6245203dd9bb46a18e8f3b4526d16b82" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } } }
I20240503 07:58:45.166074 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet 52c02ae315844d90aa9454b9c022f27d with delete_type TABLET_DATA_TOMBSTONED (TS 80435ecadfcf4ede849c7a97abea952d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59086
I20240503 07:58:45.166635 5297 tablet_replica.cc:330] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d: stopping tablet replica
I20240503 07:58:45.167001 5297 raft_consensus.cc:2229] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:45.167191 5297 raft_consensus.cc:2258] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:45.168381 5297 ts_tablet_manager.cc:1875] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:45.171000 5297 ts_tablet_manager.cc:1888] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 07:58:45.171108 5297 log.cc:1196] T 52c02ae315844d90aa9454b9c022f27d P 80435ecadfcf4ede849c7a97abea952d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/52c02ae315844d90aa9454b9c022f27d
I20240503 07:58:45.171766 2251 catalog_manager.cc:4830] TS 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895): tablet 52c02ae315844d90aa9454b9c022f27d (table rebalance_test_table_4 [id=76719076b11844e4b721cc235f022423]) successfully deleted
I20240503 07:58:45.224519 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.226830 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.227882 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.231982 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.232555 2537 raft_consensus.cc:1261] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Refusing update from remote peer 27715b19d4ad40d89c2d12e66aca6a34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:58:45.233237 5011 consensus_queue.cc:1035] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:45.245044 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.249846 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.253934 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.297807 3818 consensus_queue.cc:237] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } }
I20240503 07:58:45.298982 5045 raft_consensus.cc:2932] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9405f157903141179a86c810396a416b (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } } }
I20240503 07:58:45.302419 2264 catalog_manager.cc:5484] T 527a672a8bd44251a56c5e47cc068e93 P 6fa4f16dd2a545dbb01a0b260ce68744 reported cstate change: config changed from index 4 to 6, VOTER 9405f157903141179a86c810396a416b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } } }
I20240503 07:58:45.308080 2385 tablet_service.cc:1514] Processing DeleteTablet for tablet 527a672a8bd44251a56c5e47cc068e93 with delete_type TABLET_DATA_TOMBSTONED (TS 9405f157903141179a86c810396a416b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39710
I20240503 07:58:45.308749 5356 tablet_replica.cc:330] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b: stopping tablet replica
I20240503 07:58:45.309144 5356 raft_consensus.cc:2229] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:45.309389 5356 raft_consensus.cc:2258] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:45.310696 5356 ts_tablet_manager.cc:1875] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:45.313195 5356 ts_tablet_manager.cc:1888] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 07:58:45.313323 5356 log.cc:1196] T 527a672a8bd44251a56c5e47cc068e93 P 9405f157903141179a86c810396a416b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/527a672a8bd44251a56c5e47cc068e93
I20240503 07:58:45.314055 2248 catalog_manager.cc:4830] TS 9405f157903141179a86c810396a416b (127.0.99.129:35381): tablet 527a672a8bd44251a56c5e47cc068e93 (table rebalance_test_table_1 [id=078c8b9967d94522a24e2b3bf16902cf]) successfully deleted
I20240503 07:58:45.340977 2537 raft_consensus.cc:1261] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Refusing update from remote peer e4a21231285448c88ef7465d265f20b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:58:45.341643 5237 consensus_queue.cc:1035] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:45.354410 2406 raft_consensus.cc:1261] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Refusing update from remote peer 1147cda1821744448cfea2164c8f3d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:58:45.354938 4918 consensus_queue.cc:1035] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:45.400090 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.400763 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.401417 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.405097 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.427013 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.427104 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.452757 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.491469 3818 consensus_queue.cc:237] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } }
I20240503 07:58:45.492305 5045 raft_consensus.cc:2932] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9405f157903141179a86c810396a416b (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } } }
I20240503 07:58:45.494771 2261 catalog_manager.cc:5484] T 5031a2c9d5af4289928e0e43488931f8 P 6fa4f16dd2a545dbb01a0b260ce68744 reported cstate change: config changed from index 4 to 6, VOTER 9405f157903141179a86c810396a416b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fa4f16dd2a545dbb01a0b260ce68744" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43115 } } }
I20240503 07:58:45.497737 2385 tablet_service.cc:1514] Processing DeleteTablet for tablet 5031a2c9d5af4289928e0e43488931f8 with delete_type TABLET_DATA_TOMBSTONED (TS 9405f157903141179a86c810396a416b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39710
I20240503 07:58:45.498013 5356 tablet_replica.cc:330] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b: stopping tablet replica
I20240503 07:58:45.498199 5356 raft_consensus.cc:2229] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:45.498389 5356 raft_consensus.cc:2258] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:45.498961 5356 ts_tablet_manager.cc:1875] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:45.501835 5356 ts_tablet_manager.cc:1888] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 07:58:45.501924 5356 log.cc:1196] T 5031a2c9d5af4289928e0e43488931f8 P 9405f157903141179a86c810396a416b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5031a2c9d5af4289928e0e43488931f8
I20240503 07:58:45.502477 2248 catalog_manager.cc:4830] TS 9405f157903141179a86c810396a416b (127.0.99.129:35381): tablet 5031a2c9d5af4289928e0e43488931f8 (table rebalance_test_table_2 [id=1bdfc2dd24fe4ed6bae2f15170340978]) successfully deleted
I20240503 07:58:45.558957 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.559381 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.561125 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.567767 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.569473 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.579085 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.581393 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.629963 3687 consensus_queue.cc:237] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } }
I20240503 07:58:45.631049 5376 raft_consensus.cc:2932] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9405f157903141179a86c810396a416b (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } } }
I20240503 07:58:45.633774 2260 catalog_manager.cc:5484] T dea872e4b037445c93067135349f57f4 P 1147cda1821744448cfea2164c8f3d58 reported cstate change: config changed from index 4 to 6, VOTER 9405f157903141179a86c810396a416b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "1147cda1821744448cfea2164c8f3d58" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } } }
I20240503 07:58:45.636894 2385 tablet_service.cc:1514] Processing DeleteTablet for tablet dea872e4b037445c93067135349f57f4 with delete_type TABLET_DATA_TOMBSTONED (TS 9405f157903141179a86c810396a416b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39710
I20240503 07:58:45.637359 5356 tablet_replica.cc:330] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b: stopping tablet replica
I20240503 07:58:45.637527 5356 raft_consensus.cc:2229] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:45.637672 5356 raft_consensus.cc:2258] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:45.638206 5356 ts_tablet_manager.cc:1875] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:45.640712 5356 ts_tablet_manager.cc:1888] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 07:58:45.640801 5356 log.cc:1196] T dea872e4b037445c93067135349f57f4 P 9405f157903141179a86c810396a416b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/dea872e4b037445c93067135349f57f4
I20240503 07:58:45.641376 2248 catalog_manager.cc:4830] TS 9405f157903141179a86c810396a416b (127.0.99.129:35381): tablet dea872e4b037445c93067135349f57f4 (table rebalance_test_table_4 [id=76719076b11844e4b721cc235f022423]) successfully deleted
I20240503 07:58:45.698778 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.701452 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.702008 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.703482 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.705920 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.708402 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.721592 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.772675 3949 consensus_queue.cc:237] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } }
I20240503 07:58:45.773525 5189 raft_consensus.cc:2932] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } } }
I20240503 07:58:45.775977 2257 catalog_manager.cc:5484] T e942e5928aab48eb9944179c292b8d1b P 6245203dd9bb46a18e8f3b4526d16b82 reported cstate change: config changed from index 4 to 6, VOTER 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "6245203dd9bb46a18e8f3b4526d16b82" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6245203dd9bb46a18e8f3b4526d16b82" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46183 } } }
I20240503 07:58:45.778796 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet e942e5928aab48eb9944179c292b8d1b with delete_type TABLET_DATA_TOMBSTONED (TS 80435ecadfcf4ede849c7a97abea952d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59086
I20240503 07:58:45.779140 5524 tablet_replica.cc:330] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d: stopping tablet replica
I20240503 07:58:45.779284 5524 raft_consensus.cc:2229] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:45.779397 5524 raft_consensus.cc:2258] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:45.779968 5524 ts_tablet_manager.cc:1875] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:45.782153 5524 ts_tablet_manager.cc:1888] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 07:58:45.782238 5524 log.cc:1196] T e942e5928aab48eb9944179c292b8d1b P 80435ecadfcf4ede849c7a97abea952d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e942e5928aab48eb9944179c292b8d1b
I20240503 07:58:45.783118 2251 catalog_manager.cc:4830] TS 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895): tablet e942e5928aab48eb9944179c292b8d1b (table rebalance_test_table_1 [id=078c8b9967d94522a24e2b3bf16902cf]) successfully deleted
I20240503 07:58:45.840004 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.842151 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.844087 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.845360 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.845933 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.850579 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.859490 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.913229 4080 consensus_queue.cc:237] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } }
I20240503 07:58:45.914216 5010 raft_consensus.cc:2932] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9405f157903141179a86c810396a416b (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } } }
I20240503 07:58:45.918067 2264 catalog_manager.cc:5484] T 1246a3d482304f02bbfe1c97c2e5a1ad P 27715b19d4ad40d89c2d12e66aca6a34 reported cstate change: config changed from index 4 to 6, VOTER 9405f157903141179a86c810396a416b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "27715b19d4ad40d89c2d12e66aca6a34" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } } }
I20240503 07:58:45.920878 2385 tablet_service.cc:1514] Processing DeleteTablet for tablet 1246a3d482304f02bbfe1c97c2e5a1ad with delete_type TABLET_DATA_TOMBSTONED (TS 9405f157903141179a86c810396a416b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39710
I20240503 07:58:45.921123 5356 tablet_replica.cc:330] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b: stopping tablet replica
I20240503 07:58:45.921265 5356 raft_consensus.cc:2229] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:45.921386 5356 raft_consensus.cc:2258] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:45.921897 5356 ts_tablet_manager.cc:1875] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:45.923903 5356 ts_tablet_manager.cc:1888] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 07:58:45.923982 5356 log.cc:1196] T 1246a3d482304f02bbfe1c97c2e5a1ad P 9405f157903141179a86c810396a416b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1246a3d482304f02bbfe1c97c2e5a1ad
I20240503 07:58:45.924531 2248 catalog_manager.cc:4830] TS 9405f157903141179a86c810396a416b (127.0.99.129:35381): tablet 1246a3d482304f02bbfe1c97c2e5a1ad (table rebalance_test_table_0 [id=5311ba7c92bf4c08b8ac5d9435e4950f]) successfully deleted
I20240503 07:58:45.984439 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.984480 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:45.991238 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:45.999282 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:46.000135 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.018571 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.028030 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.066336 3687 consensus_queue.cc:237] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } }
I20240503 07:58:46.067013 5376 raft_consensus.cc:2932] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9405f157903141179a86c810396a416b (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } } }
I20240503 07:58:46.069370 2264 catalog_manager.cc:5484] T e9e176777a54493f94cdf609210874b6 P 1147cda1821744448cfea2164c8f3d58 reported cstate change: config changed from index 4 to 6, VOTER 9405f157903141179a86c810396a416b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "1147cda1821744448cfea2164c8f3d58" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1147cda1821744448cfea2164c8f3d58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46551 } } }
I20240503 07:58:46.071943 2385 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e176777a54493f94cdf609210874b6 with delete_type TABLET_DATA_TOMBSTONED (TS 9405f157903141179a86c810396a416b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39710
I20240503 07:58:46.072175 5356 tablet_replica.cc:330] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b: stopping tablet replica
I20240503 07:58:46.072302 5356 raft_consensus.cc:2229] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:46.072410 5356 raft_consensus.cc:2258] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:46.073129 5356 ts_tablet_manager.cc:1875] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:46.075201 5356 ts_tablet_manager.cc:1888] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 07:58:46.075276 5356 log.cc:1196] T e9e176777a54493f94cdf609210874b6 P 9405f157903141179a86c810396a416b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e9e176777a54493f94cdf609210874b6
I20240503 07:58:46.075757 2248 catalog_manager.cc:4830] TS 9405f157903141179a86c810396a416b (127.0.99.129:35381): tablet e9e176777a54493f94cdf609210874b6 (table rebalance_test_table_2 [id=1bdfc2dd24fe4ed6bae2f15170340978]) successfully deleted
I20240503 07:58:46.128953 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:46.132114 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.133387 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.134171 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.135674 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.143674 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:46.149163 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.202834 4080 consensus_queue.cc:237] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } }
I20240503 07:58:46.203495 5010 raft_consensus.cc:2932] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } } }
I20240503 07:58:46.205999 2261 catalog_manager.cc:5484] T 6ed9251d58be42d3841e8eb72b520353 P 27715b19d4ad40d89c2d12e66aca6a34 reported cstate change: config changed from index 4 to 6, VOTER 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "27715b19d4ad40d89c2d12e66aca6a34" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27715b19d4ad40d89c2d12e66aca6a34" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34969 } } }
I20240503 07:58:46.208959 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ed9251d58be42d3841e8eb72b520353 with delete_type TABLET_DATA_TOMBSTONED (TS 80435ecadfcf4ede849c7a97abea952d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59086
I20240503 07:58:46.209195 5524 tablet_replica.cc:330] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d: stopping tablet replica
I20240503 07:58:46.209332 5524 raft_consensus.cc:2229] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:46.209456 5524 raft_consensus.cc:2258] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:46.210186 5524 ts_tablet_manager.cc:1875] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:46.212314 5524 ts_tablet_manager.cc:1888] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 07:58:46.212388 5524 log.cc:1196] T 6ed9251d58be42d3841e8eb72b520353 P 80435ecadfcf4ede849c7a97abea952d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6ed9251d58be42d3841e8eb72b520353
I20240503 07:58:46.212852 2251 catalog_manager.cc:4830] TS 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895): tablet 6ed9251d58be42d3841e8eb72b520353 (table rebalance_test_table_3 [id=28225e90d3cd4d9e9330eaeb67f6f2a1]) successfully deleted
I20240503 07:58:46.267969 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.268885 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.269948 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.271097 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:46.273130 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:46.283617 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.306530 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.342451 3550 consensus_queue.cc:237] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } }
I20240503 07:58:46.343322 5730 raft_consensus.cc:2932] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } } }
I20240503 07:58:46.346045 2257 catalog_manager.cc:5484] T 12be85cd12b84b4aa6389177679cd9d3 P e4a21231285448c88ef7465d265f20b5 reported cstate change: config changed from index 4 to 6, VOTER 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "e4a21231285448c88ef7465d265f20b5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4a21231285448c88ef7465d265f20b5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38167 } } }
I20240503 07:58:46.348646 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet 12be85cd12b84b4aa6389177679cd9d3 with delete_type TABLET_DATA_TOMBSTONED (TS 80435ecadfcf4ede849c7a97abea952d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59086
I20240503 07:58:46.348904 5524 tablet_replica.cc:330] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d: stopping tablet replica
I20240503 07:58:46.349056 5524 raft_consensus.cc:2229] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:46.349165 5524 raft_consensus.cc:2258] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:46.349694 5524 ts_tablet_manager.cc:1875] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:46.352118 5524 ts_tablet_manager.cc:1888] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 07:58:46.352208 5524 log.cc:1196] T 12be85cd12b84b4aa6389177679cd9d3 P 80435ecadfcf4ede849c7a97abea952d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/12be85cd12b84b4aa6389177679cd9d3
I20240503 07:58:46.352708 2251 catalog_manager.cc:4830] TS 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895): tablet 12be85cd12b84b4aa6389177679cd9d3 (table rebalance_test_table_3 [id=28225e90d3cd4d9e9330eaeb67f6f2a1]) successfully deleted
I20240503 07:58:46.661564 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.661924 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.665402 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:46.666775 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.675673 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.690167 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.691020 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:46.784746 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.785087 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.785174 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.787609 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.796334 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:46.804538 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:46.811601 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:47.060111 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.060143 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.064110 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:47.066232 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:47.074837 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.074867 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.103338 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.209061 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.209524 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.210111 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.217672 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:47.218175 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.220153 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.221676 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:47.331948 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.332964 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.333835 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.335240 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.339125 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:47.340668 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.361208 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6b1bd3a95fbd45e58255ec3143f75686 | 127.0.99.190:34437 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:44085 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1147cda1821744448cfea2164c8f3d58 | 127.0.99.132:46551 | HEALTHY | <none> | 2 | 0
27715b19d4ad40d89c2d12e66aca6a34 | 127.0.99.135:34969 | HEALTHY | <none> | 2 | 0
6245203dd9bb46a18e8f3b4526d16b82 | 127.0.99.134:46183 | HEALTHY | <none> | 2 | 0
6fa4f16dd2a545dbb01a0b260ce68744 | 127.0.99.133:43115 | HEALTHY | <none> | 2 | 0
80435ecadfcf4ede849c7a97abea952d | 127.0.99.130:33895 | HEALTHY | <none> | 3 | 0
9405f157903141179a86c810396a416b | 127.0.99.129:35381 | HEALTHY | <none> | 3 | 0
e4a21231285448c88ef7465d265f20b5 | 127.0.99.131:38167 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:35381
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:33895
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:38167
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:46551
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:43115
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:46183
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:34969
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:35381
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:33895
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:38167
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:46551
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:43115
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:46183
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:34969
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:44085 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 07:58:47.391788 398 log_verifier.cc:126] Checking tablet 0189e0b9411f41908cbce67fce3ea308
I20240503 07:58:47.401949 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 0189e0b9411f41908cbce67fce3ea308
I20240503 07:58:47.402056 398 log_verifier.cc:126] Checking tablet 1246a3d482304f02bbfe1c97c2e5a1ad
I20240503 07:58:47.411633 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 1246a3d482304f02bbfe1c97c2e5a1ad
I20240503 07:58:47.411726 398 log_verifier.cc:126] Checking tablet 12be85cd12b84b4aa6389177679cd9d3
I20240503 07:58:47.421778 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 12be85cd12b84b4aa6389177679cd9d3
I20240503 07:58:47.421867 398 log_verifier.cc:126] Checking tablet 403bf0824c2245a19f125db609649afd
I20240503 07:58:47.432046 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 403bf0824c2245a19f125db609649afd
I20240503 07:58:47.432139 398 log_verifier.cc:126] Checking tablet 5031a2c9d5af4289928e0e43488931f8
I20240503 07:58:47.442425 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 5031a2c9d5af4289928e0e43488931f8
I20240503 07:58:47.442498 398 log_verifier.cc:126] Checking tablet 527a672a8bd44251a56c5e47cc068e93
I20240503 07:58:47.452661 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 527a672a8bd44251a56c5e47cc068e93
I20240503 07:58:47.452737 398 log_verifier.cc:126] Checking tablet 52c02ae315844d90aa9454b9c022f27d
I20240503 07:58:47.463235 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 52c02ae315844d90aa9454b9c022f27d
I20240503 07:58:47.463322 398 log_verifier.cc:126] Checking tablet 6ed9251d58be42d3841e8eb72b520353
I20240503 07:58:47.473651 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6ed9251d58be42d3841e8eb72b520353
I20240503 07:58:47.473737 398 log_verifier.cc:126] Checking tablet 9841efae7c614c8d826151a88a1c5c6f
I20240503 07:58:47.483697 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9841efae7c614c8d826151a88a1c5c6f
I20240503 07:58:47.483773 398 log_verifier.cc:126] Checking tablet b46fc192e7aa439d988ee7c4e51603c2
I20240503 07:58:47.493398 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b46fc192e7aa439d988ee7c4e51603c2
I20240503 07:58:47.493471 398 log_verifier.cc:126] Checking tablet d30b596efc8f4f2190a6ca4eba17640b
I20240503 07:58:47.503170 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d30b596efc8f4f2190a6ca4eba17640b
I20240503 07:58:47.503244 398 log_verifier.cc:126] Checking tablet d81db8a9c3da42e7a26ef1c665b6c93b
I20240503 07:58:47.513218 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d81db8a9c3da42e7a26ef1c665b6c93b
I20240503 07:58:47.513298 398 log_verifier.cc:126] Checking tablet dea872e4b037445c93067135349f57f4
I20240503 07:58:47.523131 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet dea872e4b037445c93067135349f57f4
I20240503 07:58:47.523211 398 log_verifier.cc:126] Checking tablet e942e5928aab48eb9944179c292b8d1b
I20240503 07:58:47.532663 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e942e5928aab48eb9944179c292b8d1b
I20240503 07:58:47.532742 398 log_verifier.cc:126] Checking tablet e9e176777a54493f94cdf609210874b6
I20240503 07:58:47.542402 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e9e176777a54493f94cdf609210874b6
I20240503 07:58:47.542469 398 log_verifier.cc:126] Checking tablet ef2c306c822246ef949e287a9937d94d
I20240503 07:58:47.551832 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ef2c306c822246ef949e287a9937d94d
I20240503 07:58:47.554087 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--builtin_ntp_servers=127.0.99.148:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 07:58:47.673758 5947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:47.674003 5947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:47.674052 5947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:47.677435 5947 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:47.677528 5947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:47.677606 5947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:58:47.681718 5947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34437
--never_fsync=true
--heap_profile_path=/tmp/kudu.5947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:47.682885 5947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:47.684140 5947 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:47.693856 5955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:47.694308 5953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:47.694876 5947 server_base.cc:1034] running on GCE node
W20240503 07:58:47.695293 5952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:47.695766 5947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:47.696465 5947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:47.697633 5947 hybrid_clock.cc:648] HybridClock initialized: now 1714723127697609 us; error 41 us; skew 500 ppm
I20240503 07:58:47.699868 5947 webserver.cc:460] Webserver started at http://127.0.99.136:35681/ using document root <none> and password file <none>
I20240503 07:58:47.700501 5947 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:47.700565 5947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:47.700752 5947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:47.702772 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "eba1d577c2234579ba4f4b7b42dabb0c"
format_stamp: "Formatted at 2024-05-03 07:58:47 on dist-test-slave-bj8b"
I20240503 07:58:47.703282 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "eba1d577c2234579ba4f4b7b42dabb0c"
format_stamp: "Formatted at 2024-05-03 07:58:47 on dist-test-slave-bj8b"
I20240503 07:58:47.707016 5947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:47.709553 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:47.710800 5947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:47.710935 5947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "eba1d577c2234579ba4f4b7b42dabb0c"
format_stamp: "Formatted at 2024-05-03 07:58:47 on dist-test-slave-bj8b"
I20240503 07:58:47.711040 5947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:47.754726 5947 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:47.755522 5947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:47.755703 5947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:47.756325 5947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:47.757372 5947 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:47.757427 5947 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:47.757488 5947 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:47.757512 5947 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:47.766960 5947 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:38053
I20240503 07:58:47.767037 6074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:38053 every 8 connection(s)
I20240503 07:58:47.768013 5947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:58:47.770823 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 5947
I20240503 07:58:47.770951 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 07:58:47.777870 6075 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34437
I20240503 07:58:47.778131 6075 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:47.778738 6075 heartbeater.cc:511] Master 127.0.99.190:34437 requested a full tablet report, sending...
I20240503 07:58:47.779891 2264 ts_manager.cc:194] Registered new tserver with Master: eba1d577c2234579ba4f4b7b42dabb0c (127.0.99.136:38053)
I20240503 07:58:47.780536 2264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:40955
I20240503 07:58:47.984087 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.988263 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.988515 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:47.998190 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:47.999827 6009 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:48.013974 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.014624 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.026218 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.136577 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.138223 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.142570 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.146711 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.149721 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.155225 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.155279 6009 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:48.175907 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.218808 2536 consensus_queue.cc:237] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } }
W20240503 07:58:48.223726 2473 consensus_peers.cc:485] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d -> Peer eba1d577c2234579ba4f4b7b42dabb0c (127.0.99.136:38053): Couldn't send request to peer eba1d577c2234579ba4f4b7b42dabb0c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9841efae7c614c8d826151a88a1c5c6f. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:48.783154 6075 heartbeater.cc:503] Master 127.0.99.190:34437 was elected leader, sending a full tablet report...
I20240503 07:58:48.807868 6181 ts_tablet_manager.cc:902] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c: Initiating tablet copy from peer 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895)
I20240503 07:58:48.808602 6181 tablet_copy_client.cc:308] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33895
I20240503 07:58:48.812512 2557 tablet_copy_service.cc:140] P 80435ecadfcf4ede849c7a97abea952d: Received BeginTabletCopySession request for tablet 9841efae7c614c8d826151a88a1c5c6f from peer eba1d577c2234579ba4f4b7b42dabb0c ({username='slave'} at 127.0.99.136:42853)
I20240503 07:58:48.812634 2557 tablet_copy_service.cc:161] P 80435ecadfcf4ede849c7a97abea952d: Beginning new tablet copy session on tablet 9841efae7c614c8d826151a88a1c5c6f from peer eba1d577c2234579ba4f4b7b42dabb0c at {username='slave'} at 127.0.99.136:42853: session id = eba1d577c2234579ba4f4b7b42dabb0c-9841efae7c614c8d826151a88a1c5c6f
I20240503 07:58:48.813441 2557 tablet_copy_source_session.cc:205] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:48.817080 6181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9841efae7c614c8d826151a88a1c5c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:48.822939 6181 tablet_copy_client.cc:783] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:48.823189 6181 tablet_copy_client.cc:647] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:48.824908 6181 tablet_copy_client.cc:523] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:48.826911 6181 tablet_bootstrap.cc:492] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c: Bootstrap starting.
I20240503 07:58:48.832834 6181 log.cc:824] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:48.854262 6181 tablet_bootstrap.cc:492] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:58:48.854755 6181 tablet_bootstrap.cc:492] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c: Bootstrap complete.
I20240503 07:58:48.855118 6181 ts_tablet_manager.cc:1367] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c: Time spent bootstrapping tablet: real 0.028s user 0.024s sys 0.003s
I20240503 07:58:48.857491 6181 raft_consensus.cc:348] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } }
I20240503 07:58:48.858225 6181 raft_consensus.cc:729] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba1d577c2234579ba4f4b7b42dabb0c, State: Initialized, Role: FOLLOWER
I20240503 07:58:48.858600 6181 consensus_queue.cc:260] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } }
I20240503 07:58:48.859503 6181 ts_tablet_manager.cc:1398] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:58:48.860389 2557 tablet_copy_service.cc:342] P 80435ecadfcf4ede849c7a97abea952d: Request end of tablet copy session eba1d577c2234579ba4f4b7b42dabb0c-9841efae7c614c8d826151a88a1c5c6f received from {username='slave'} at 127.0.99.136:42853
I20240503 07:58:48.860496 2557 tablet_copy_service.cc:433] P 80435ecadfcf4ede849c7a97abea952d: ending tablet copy session eba1d577c2234579ba4f4b7b42dabb0c-9841efae7c614c8d826151a88a1c5c6f on tablet 9841efae7c614c8d826151a88a1c5c6f with peer eba1d577c2234579ba4f4b7b42dabb0c
I20240503 07:58:49.312944 6029 raft_consensus.cc:1203] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:49.314390 6191 raft_consensus.cc:2932] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eba1d577c2234579ba4f4b7b42dabb0c (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } } }
I20240503 07:58:49.315089 6029 raft_consensus.cc:2932] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eba1d577c2234579ba4f4b7b42dabb0c (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } } }
I20240503 07:58:49.315904 6191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:49242 (request call id 0) took 1097 ms. Trace:
I20240503 07:58:49.316035 6191 rpcz_store.cc:276] 0503 07:58:48.218436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 07:58:48.218482 (+ 46us) service_pool.cc:225] Handling call
0503 07:58:48.219329 (+ 847us) log.cc:842] Serialized 257 byte log entry
0503 07:58:49.315884 (+1096555us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":61,"raft.queue_time_us":100,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"thread_start_us":123,"threads_started":2,"wal-append.queue_time_us":157,"wal-append.run_cpu_time_us":345,"wal-append.run_wall_time_us":1000416}
I20240503 07:58:49.318710 2256 catalog_manager.cc:5484] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c reported cstate change: config changed from index -1 to 4, VOTER eba1d577c2234579ba4f4b7b42dabb0c (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "80435ecadfcf4ede849c7a97abea952d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } } }
I20240503 07:58:49.336231 2406 consensus_queue.cc:237] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } }
W20240503 07:58:49.340799 2342 consensus_peers.cc:485] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b -> Peer eba1d577c2234579ba4f4b7b42dabb0c (127.0.99.136:38053): Couldn't send request to peer eba1d577c2234579ba4f4b7b42dabb0c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef2c306c822246ef949e287a9937d94d. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:49.769256 6219 ts_tablet_manager.cc:902] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c: Initiating tablet copy from peer 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:49.769817 6219 tablet_copy_client.cc:308] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35381
I20240503 07:58:49.774176 2426 tablet_copy_service.cc:140] P 9405f157903141179a86c810396a416b: Received BeginTabletCopySession request for tablet ef2c306c822246ef949e287a9937d94d from peer eba1d577c2234579ba4f4b7b42dabb0c ({username='slave'} at 127.0.99.136:47049)
I20240503 07:58:49.774333 2426 tablet_copy_service.cc:161] P 9405f157903141179a86c810396a416b: Beginning new tablet copy session on tablet ef2c306c822246ef949e287a9937d94d from peer eba1d577c2234579ba4f4b7b42dabb0c at {username='slave'} at 127.0.99.136:47049: session id = eba1d577c2234579ba4f4b7b42dabb0c-ef2c306c822246ef949e287a9937d94d
I20240503 07:58:49.775151 2426 tablet_copy_source_session.cc:205] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:49.776093 6219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2c306c822246ef949e287a9937d94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:49.778007 6219 tablet_copy_client.cc:783] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:49.778162 6219 tablet_copy_client.cc:647] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:49.779511 6219 tablet_copy_client.cc:523] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:49.780731 6219 tablet_bootstrap.cc:492] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c: Bootstrap starting.
I20240503 07:58:49.798916 6219 tablet_bootstrap.cc:492] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:58:49.799284 6219 tablet_bootstrap.cc:492] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c: Bootstrap complete.
I20240503 07:58:49.799537 6219 ts_tablet_manager.cc:1367] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
I20240503 07:58:49.800148 6219 raft_consensus.cc:348] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } }
I20240503 07:58:49.800401 6219 raft_consensus.cc:729] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba1d577c2234579ba4f4b7b42dabb0c, State: Initialized, Role: FOLLOWER
I20240503 07:58:49.800509 6219 consensus_queue.cc:260] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } }
I20240503 07:58:49.800766 6219 ts_tablet_manager.cc:1398] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:49.801460 2426 tablet_copy_service.cc:342] P 9405f157903141179a86c810396a416b: Request end of tablet copy session eba1d577c2234579ba4f4b7b42dabb0c-ef2c306c822246ef949e287a9937d94d received from {username='slave'} at 127.0.99.136:47049
I20240503 07:58:49.801544 2426 tablet_copy_service.cc:433] P 9405f157903141179a86c810396a416b: ending tablet copy session eba1d577c2234579ba4f4b7b42dabb0c-ef2c306c822246ef949e287a9937d94d on tablet ef2c306c822246ef949e287a9937d94d with peer eba1d577c2234579ba4f4b7b42dabb0c
I20240503 07:58:50.269035 6029 raft_consensus.cc:1203] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:50.269981 6227 raft_consensus.cc:2932] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eba1d577c2234579ba4f4b7b42dabb0c (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } } }
I20240503 07:58:50.270498 6029 raft_consensus.cc:2932] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eba1d577c2234579ba4f4b7b42dabb0c (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } } }
I20240503 07:58:50.272964 2264 catalog_manager.cc:5484] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b reported cstate change: config changed from index -1 to 4, VOTER eba1d577c2234579ba4f4b7b42dabb0c (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "9405f157903141179a86c810396a416b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } } }
I20240503 07:58:50.331413 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.335904 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:50.338954 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.339886 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.340735 6009 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:50.354424 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:50.354519 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.369647 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.398515 2406 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef2c306c822246ef949e287a9937d94d"
dest_uuid: "9405f157903141179a86c810396a416b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49826
I20240503 07:58:50.398684 2406 raft_consensus.cc:595] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:50.456807 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.458654 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.462399 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:50.464982 6009 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:50.475500 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:50.476855 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.483407 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.495121 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.579473 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.580458 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.581580 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.582463 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.582623 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:50.590548 6009 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:50.592533 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:50.593926 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.639439 2536 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9841efae7c614c8d826151a88a1c5c6f"
dest_uuid: "80435ecadfcf4ede849c7a97abea952d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49290
I20240503 07:58:50.639627 2536 raft_consensus.cc:595] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:50.699388 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.699385 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.708041 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.708761 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:50.716387 2517 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:50.716416 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.737203 6009 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:50.738912 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:50.893411 6395 raft_consensus.cc:979] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b: : Instructing follower eba1d577c2234579ba4f4b7b42dabb0c to start an election
I20240503 07:58:50.893519 6395 raft_consensus.cc:1067] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 1 LEADER]: Signalling peer eba1d577c2234579ba4f4b7b42dabb0c to start an election
I20240503 07:58:50.894052 6029 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef2c306c822246ef949e287a9937d94d"
dest_uuid: "eba1d577c2234579ba4f4b7b42dabb0c"
from {username='slave'} at 127.0.99.129:56013
I20240503 07:58:50.894282 6029 raft_consensus.cc:482] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:50.894436 6029 raft_consensus.cc:3037] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:50.895350 6029 raft_consensus.cc:504] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } }
I20240503 07:58:50.896371 6029 leader_election.cc:288] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [CANDIDATE]: Term 2 election: Requested vote from peers 9405f157903141179a86c810396a416b (127.0.99.129:35381)
I20240503 07:58:50.896831 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2c306c822246ef949e287a9937d94d" candidate_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9405f157903141179a86c810396a416b"
I20240503 07:58:50.896977 2406 raft_consensus.cc:3032] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:50.897013 2406 raft_consensus.cc:729] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9405f157903141179a86c810396a416b, State: Running, Role: LEADER
I20240503 07:58:50.897141 2406 consensus_queue.cc:260] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } }
I20240503 07:58:50.897260 2406 raft_consensus.cc:3037] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:50.898072 2406 raft_consensus.cc:2454] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eba1d577c2234579ba4f4b7b42dabb0c in term 2.
I20240503 07:58:50.898526 5962 leader_election.cc:302] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9405f157903141179a86c810396a416b, eba1d577c2234579ba4f4b7b42dabb0c; no voters:
I20240503 07:58:50.898873 6397 raft_consensus.cc:2781] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:50.899048 6397 raft_consensus.cc:686] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [term 2 LEADER]: Becoming Leader. State: Replica: eba1d577c2234579ba4f4b7b42dabb0c, State: Running, Role: LEADER
I20240503 07:58:50.899402 6397 consensus_queue.cc:237] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } }
I20240503 07:58:50.901746 2261 catalog_manager.cc:5484] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c reported cstate change: term changed from 1 to 2, leader changed from 9405f157903141179a86c810396a416b (127.0.99.129) to eba1d577c2234579ba4f4b7b42dabb0c (127.0.99.136). New cstate: current_term: 2 leader_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } } }
I20240503 07:58:50.908424 6295 raft_consensus.cc:979] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d: : Instructing follower eba1d577c2234579ba4f4b7b42dabb0c to start an election
I20240503 07:58:50.908548 6295 raft_consensus.cc:1067] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Signalling peer eba1d577c2234579ba4f4b7b42dabb0c to start an election
I20240503 07:58:50.909061 6029 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9841efae7c614c8d826151a88a1c5c6f"
dest_uuid: "eba1d577c2234579ba4f4b7b42dabb0c"
from {username='slave'} at 127.0.99.130:48787
I20240503 07:58:50.909217 6029 raft_consensus.cc:482] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:50.909279 6029 raft_consensus.cc:3037] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:50.910145 6029 raft_consensus.cc:504] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } }
I20240503 07:58:50.910584 6029 leader_election.cc:288] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [CANDIDATE]: Term 2 election: Requested vote from peers 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895)
I20240503 07:58:50.911034 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9841efae7c614c8d826151a88a1c5c6f" candidate_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "80435ecadfcf4ede849c7a97abea952d"
I20240503 07:58:50.911175 2536 raft_consensus.cc:3032] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:50.911222 2536 raft_consensus.cc:729] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80435ecadfcf4ede849c7a97abea952d, State: Running, Role: LEADER
I20240503 07:58:50.911373 2536 consensus_queue.cc:260] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } }
I20240503 07:58:50.911522 2536 raft_consensus.cc:3037] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:50.913479 2536 raft_consensus.cc:2454] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eba1d577c2234579ba4f4b7b42dabb0c in term 2.
I20240503 07:58:50.913901 5965 leader_election.cc:302] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 80435ecadfcf4ede849c7a97abea952d, eba1d577c2234579ba4f4b7b42dabb0c; no voters:
I20240503 07:58:50.914093 6397 raft_consensus.cc:2781] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:50.914217 6397 raft_consensus.cc:686] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [term 2 LEADER]: Becoming Leader. State: Replica: eba1d577c2234579ba4f4b7b42dabb0c, State: Running, Role: LEADER
I20240503 07:58:50.914398 6397 consensus_queue.cc:237] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } }
I20240503 07:58:50.916316 2261 catalog_manager.cc:5484] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c reported cstate change: term changed from 1 to 2, leader changed from 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) to eba1d577c2234579ba4f4b7b42dabb0c (127.0.99.136). New cstate: current_term: 2 leader_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 } } peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } } }
I20240503 07:58:51.025717 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.028124 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.029435 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.031425 2385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.042446 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.047494 2517 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.054153 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.058594 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.144975 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.147257 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.148185 2385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.156358 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.157828 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.159411 2517 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.171077 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.197957 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.448582 2536 raft_consensus.cc:1261] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Refusing update from remote peer eba1d577c2234579ba4f4b7b42dabb0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:58:51.449301 6496 consensus_queue.cc:1035] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "80435ecadfcf4ede849c7a97abea952d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:51.485241 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.487231 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.487505 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.488994 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.496091 2406 raft_consensus.cc:1261] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Refusing update from remote peer eba1d577c2234579ba4f4b7b42dabb0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:58:51.496733 6496 consensus_queue.cc:1035] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9405f157903141179a86c810396a416b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:51.499285 2385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.507704 2517 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.508468 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.526903 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.566533 6029 consensus_queue.cc:237] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } }
I20240503 07:58:51.567241 6499 raft_consensus.cc:2932] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9405f157903141179a86c810396a416b (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } } }
I20240503 07:58:51.570247 2256 catalog_manager.cc:5484] T ef2c306c822246ef949e287a9937d94d P eba1d577c2234579ba4f4b7b42dabb0c reported cstate change: config changed from index 4 to 6, VOTER 9405f157903141179a86c810396a416b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } } }
I20240503 07:58:51.573417 2385 tablet_service.cc:1514] Processing DeleteTablet for tablet ef2c306c822246ef949e287a9937d94d with delete_type TABLET_DATA_TOMBSTONED (TS 9405f157903141179a86c810396a416b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39710
I20240503 07:58:51.573810 6550 tablet_replica.cc:330] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b: stopping tablet replica
I20240503 07:58:51.573943 6550 raft_consensus.cc:2229] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:51.574067 6550 raft_consensus.cc:2258] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:51.575485 6550 ts_tablet_manager.cc:1875] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:51.577536 6550 ts_tablet_manager.cc:1888] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 07:58:51.577618 6550 log.cc:1196] T ef2c306c822246ef949e287a9937d94d P 9405f157903141179a86c810396a416b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ef2c306c822246ef949e287a9937d94d
I20240503 07:58:51.578195 2248 catalog_manager.cc:4830] TS 9405f157903141179a86c810396a416b (127.0.99.129:35381): tablet ef2c306c822246ef949e287a9937d94d (table rebalance_test_table_4 [id=76719076b11844e4b721cc235f022423]) successfully deleted
I20240503 07:58:51.637008 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.638243 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.638787 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.639760 2517 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.653292 2385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.654330 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.655453 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.663203 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.715121 6029 consensus_queue.cc:237] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } }
I20240503 07:58:51.715807 6499 raft_consensus.cc:2932] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } } }
I20240503 07:58:51.718767 2261 catalog_manager.cc:5484] T 9841efae7c614c8d826151a88a1c5c6f P eba1d577c2234579ba4f4b7b42dabb0c reported cstate change: config changed from index 4 to 6, VOTER 80435ecadfcf4ede849c7a97abea952d (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eba1d577c2234579ba4f4b7b42dabb0c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38053 } } }
I20240503 07:58:51.721508 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet 9841efae7c614c8d826151a88a1c5c6f with delete_type TABLET_DATA_TOMBSTONED (TS 80435ecadfcf4ede849c7a97abea952d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59086
I20240503 07:58:51.721967 6599 tablet_replica.cc:330] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d: stopping tablet replica
I20240503 07:58:51.722110 6599 raft_consensus.cc:2229] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:51.722231 6599 raft_consensus.cc:2258] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:51.723423 6599 ts_tablet_manager.cc:1875] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:51.725590 6599 ts_tablet_manager.cc:1888] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 07:58:51.725677 6599 log.cc:1196] T 9841efae7c614c8d826151a88a1c5c6f P 80435ecadfcf4ede849c7a97abea952d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9841efae7c614c8d826151a88a1c5c6f
I20240503 07:58:51.726305 2251 catalog_manager.cc:4830] TS 80435ecadfcf4ede849c7a97abea952d (127.0.99.130:33895): tablet 9841efae7c614c8d826151a88a1c5c6f (table rebalance_test_table_2 [id=1bdfc2dd24fe4ed6bae2f15170340978]) successfully deleted
I20240503 07:58:51.998026 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:51.999949 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.009905 2517 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.010089 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.010712 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.012426 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.012804 2385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.045881 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.128651 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.128756 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.132822 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.137146 2517 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.138330 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.140187 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.143599 2385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.169617 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.257836 3667 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.261127 4060 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.264957 2517 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.266544 3929 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.266534 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.266937 3798 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.267652 2385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.274875 3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6b1bd3a95fbd45e58255ec3143f75686 | 127.0.99.190:34437 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:44085 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1147cda1821744448cfea2164c8f3d58 | 127.0.99.132:46551 | HEALTHY | <none> | 2 | 0
27715b19d4ad40d89c2d12e66aca6a34 | 127.0.99.135:34969 | HEALTHY | <none> | 2 | 0
6245203dd9bb46a18e8f3b4526d16b82 | 127.0.99.134:46183 | HEALTHY | <none> | 2 | 0
6fa4f16dd2a545dbb01a0b260ce68744 | 127.0.99.133:43115 | HEALTHY | <none> | 2 | 0
80435ecadfcf4ede849c7a97abea952d | 127.0.99.130:33895 | HEALTHY | <none> | 2 | 0
9405f157903141179a86c810396a416b | 127.0.99.129:35381 | HEALTHY | <none> | 2 | 0
e4a21231285448c88ef7465d265f20b5 | 127.0.99.131:38167 | HEALTHY | <none> | 2 | 0
eba1d577c2234579ba4f4b7b42dabb0c | 127.0.99.136:38053 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:35381
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:33895
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:38167
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:46551
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:43115
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:46183
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:34969
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:38053
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:35381
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:33895
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:38167
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:46551
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:43115
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:46183
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:34969
server_dump_info_path | /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:38053
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:44085 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 07:58:52.322710 398 log_verifier.cc:126] Checking tablet 0189e0b9411f41908cbce67fce3ea308
I20240503 07:58:52.328004 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 0189e0b9411f41908cbce67fce3ea308
I20240503 07:58:52.328074 398 log_verifier.cc:126] Checking tablet 1246a3d482304f02bbfe1c97c2e5a1ad
I20240503 07:58:52.334142 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 1246a3d482304f02bbfe1c97c2e5a1ad
I20240503 07:58:52.334246 398 log_verifier.cc:126] Checking tablet 12be85cd12b84b4aa6389177679cd9d3
I20240503 07:58:52.340145 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 12be85cd12b84b4aa6389177679cd9d3
I20240503 07:58:52.340219 398 log_verifier.cc:126] Checking tablet 403bf0824c2245a19f125db609649afd
I20240503 07:58:52.346014 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 403bf0824c2245a19f125db609649afd
I20240503 07:58:52.346084 398 log_verifier.cc:126] Checking tablet 5031a2c9d5af4289928e0e43488931f8
I20240503 07:58:52.353372 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 5031a2c9d5af4289928e0e43488931f8
I20240503 07:58:52.353550 398 log_verifier.cc:126] Checking tablet 527a672a8bd44251a56c5e47cc068e93
I20240503 07:58:52.361081 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 527a672a8bd44251a56c5e47cc068e93
I20240503 07:58:52.361167 398 log_verifier.cc:126] Checking tablet 52c02ae315844d90aa9454b9c022f27d
I20240503 07:58:52.368064 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 52c02ae315844d90aa9454b9c022f27d
I20240503 07:58:52.368137 398 log_verifier.cc:126] Checking tablet 6ed9251d58be42d3841e8eb72b520353
I20240503 07:58:52.374071 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6ed9251d58be42d3841e8eb72b520353
I20240503 07:58:52.374142 398 log_verifier.cc:126] Checking tablet 9841efae7c614c8d826151a88a1c5c6f
I20240503 07:58:52.382500 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 9841efae7c614c8d826151a88a1c5c6f
I20240503 07:58:52.382573 398 log_verifier.cc:126] Checking tablet b46fc192e7aa439d988ee7c4e51603c2
I20240503 07:58:52.388305 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b46fc192e7aa439d988ee7c4e51603c2
I20240503 07:58:52.388374 398 log_verifier.cc:126] Checking tablet d30b596efc8f4f2190a6ca4eba17640b
I20240503 07:58:52.393980 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d30b596efc8f4f2190a6ca4eba17640b
I20240503 07:58:52.394042 398 log_verifier.cc:126] Checking tablet d81db8a9c3da42e7a26ef1c665b6c93b
I20240503 07:58:52.399780 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d81db8a9c3da42e7a26ef1c665b6c93b
I20240503 07:58:52.399849 398 log_verifier.cc:126] Checking tablet dea872e4b037445c93067135349f57f4
I20240503 07:58:52.405598 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet dea872e4b037445c93067135349f57f4
I20240503 07:58:52.405674 398 log_verifier.cc:126] Checking tablet e942e5928aab48eb9944179c292b8d1b
I20240503 07:58:52.411506 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e942e5928aab48eb9944179c292b8d1b
I20240503 07:58:52.411581 398 log_verifier.cc:126] Checking tablet e9e176777a54493f94cdf609210874b6
I20240503 07:58:52.417647 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e9e176777a54493f94cdf609210874b6
I20240503 07:58:52.417712 398 log_verifier.cc:126] Checking tablet ef2c306c822246ef949e287a9937d94d
I20240503 07:58:52.426136 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ef2c306c822246ef949e287a9937d94d
I20240503 07:58:52.426712 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 2324
I20240503 07:58:52.433454 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 2455
I20240503 07:58:52.439981 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 3467
I20240503 07:58:52.446045 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 3600
I20240503 07:58:52.451961 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 3736
I20240503 07:58:52.457564 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 3866
I20240503 07:58:52.465204 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 3998
I20240503 07:58:52.470662 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 5947
I20240503 07:58:52.475977 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 2233
2024-05-03T07:58:52Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (27367 ms)
[----------] 1 test from RebalanceParamTest (27367 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240503 07:58:52.511544 398 test_util.cc:268] Using random seed: 389256494
I20240503 07:58:52.512398 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:58:52.512445 398 ts_itest-base.cc:113] --------------
I20240503 07:58:52.513041 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 07:58:52.513070 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:58:52.513083 398 ts_itest-base.cc:116] --------------
2024-05-03T07:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:58:52Z Disabled control of system clock
I20240503 07:58:52.521446 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:44703
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:44703
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 07:58:52.635576 6716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:52.635864 6716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:52.635905 6716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:52.639183 6716 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:52.639281 6716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:58:52.639305 6716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:52.639328 6716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:58:52.639361 6716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:58:52.643416 6716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:44703
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:44703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:52.644526 6716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:52.645869 6716 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:52.654358 6722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:52.654733 6721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:52.655265 6724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:52.655920 6716 server_base.cc:1034] running on GCE node
I20240503 07:58:52.656562 6716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:52.657697 6716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:52.658846 6716 hybrid_clock.cc:648] HybridClock initialized: now 1714723132658828 us; error 38 us; skew 500 ppm
I20240503 07:58:52.661001 6716 webserver.cc:460] Webserver started at http://127.0.99.190:40663/ using document root <none> and password file <none>
I20240503 07:58:52.661610 6716 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:52.661669 6716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:52.661878 6716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:52.663723 6716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f72296bedf9648789ec7fcba5a38bea2"
format_stamp: "Formatted at 2024-05-03 07:58:52 on dist-test-slave-bj8b"
I20240503 07:58:52.664225 6716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f72296bedf9648789ec7fcba5a38bea2"
format_stamp: "Formatted at 2024-05-03 07:58:52 on dist-test-slave-bj8b"
I20240503 07:58:52.667868 6716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:52.670277 6730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:52.671761 6716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:52.671908 6716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f72296bedf9648789ec7fcba5a38bea2"
format_stamp: "Formatted at 2024-05-03 07:58:52 on dist-test-slave-bj8b"
I20240503 07:58:52.672025 6716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:52.701704 6716 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:52.702514 6716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:52.702692 6716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:52.709779 6716 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:44703
I20240503 07:58:52.709793 6782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:44703 every 8 connection(s)
I20240503 07:58:52.711015 6716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:58:52.714012 6783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:52.717492 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 6716
I20240503 07:58:52.717651 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:58:52.720209 6783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Bootstrap starting.
I20240503 07:58:52.722630 6783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:52.723414 6783 log.cc:824] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:52.725731 6783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: No bootstrap required, opened a new log
I20240503 07:58:52.728837 6783 raft_consensus.cc:348] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72296bedf9648789ec7fcba5a38bea2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44703 } }
I20240503 07:58:52.729053 6783 raft_consensus.cc:374] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:52.729089 6783 raft_consensus.cc:729] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72296bedf9648789ec7fcba5a38bea2, State: Initialized, Role: FOLLOWER
I20240503 07:58:52.729492 6783 consensus_queue.cc:260] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72296bedf9648789ec7fcba5a38bea2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44703 } }
I20240503 07:58:52.729609 6783 raft_consensus.cc:388] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:52.729652 6783 raft_consensus.cc:482] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:52.729743 6783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:52.730679 6783 raft_consensus.cc:504] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72296bedf9648789ec7fcba5a38bea2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44703 } }
I20240503 07:58:52.731081 6783 leader_election.cc:302] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f72296bedf9648789ec7fcba5a38bea2; no voters:
I20240503 07:58:52.731412 6783 leader_election.cc:288] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:52.731798 6788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:52.731998 6788 raft_consensus.cc:686] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [term 1 LEADER]: Becoming Leader. State: Replica: f72296bedf9648789ec7fcba5a38bea2, State: Running, Role: LEADER
I20240503 07:58:52.732308 6788 consensus_queue.cc:237] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72296bedf9648789ec7fcba5a38bea2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44703 } }
I20240503 07:58:52.732553 6783 sys_catalog.cc:564] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:58:52.733803 6789 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f72296bedf9648789ec7fcba5a38bea2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72296bedf9648789ec7fcba5a38bea2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44703 } } }
I20240503 07:58:52.733947 6789 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:52.734160 6790 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f72296bedf9648789ec7fcba5a38bea2. Latest consensus state: current_term: 1 leader_uuid: "f72296bedf9648789ec7fcba5a38bea2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72296bedf9648789ec7fcba5a38bea2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44703 } } }
I20240503 07:58:52.734283 6790 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2 [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:52.734563 6797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:58:52.737845 6797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:58:52.743263 6797 catalog_manager.cc:1347] Generated new cluster ID: c505b00f8d9243898a0590ec9d051504
I20240503 07:58:52.743355 6797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:58:52.753487 6797 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:58:52.754974 6797 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:58:52.765111 6797 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Generated new TSK 0
I20240503 07:58:52.766110 6797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:58:52.771322 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 07:58:52.893759 6807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:52.894053 6807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:52.894120 6807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:52.897550 6807 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:52.897639 6807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:52.897735 6807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:58:52.901765 6807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--never_fsync=true
--heap_profile_path=/tmp/kudu.6807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:52.902956 6807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:52.904295 6807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:52.913772 6815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:52.914280 6813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:52.914805 6812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:52.915431 6807 server_base.cc:1034] running on GCE node
I20240503 07:58:52.915987 6807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:52.916725 6807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:52.917882 6807 hybrid_clock.cc:648] HybridClock initialized: now 1714723132917865 us; error 39 us; skew 500 ppm
I20240503 07:58:52.920150 6807 webserver.cc:460] Webserver started at http://127.0.99.129:32919/ using document root <none> and password file <none>
I20240503 07:58:52.920785 6807 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:52.920851 6807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:52.921042 6807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:52.922942 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c4218904dd7e42d2be95b901a6e5db92"
format_stamp: "Formatted at 2024-05-03 07:58:52 on dist-test-slave-bj8b"
I20240503 07:58:52.923444 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c4218904dd7e42d2be95b901a6e5db92"
format_stamp: "Formatted at 2024-05-03 07:58:52 on dist-test-slave-bj8b"
I20240503 07:58:52.927209 6807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 07:58:52.929667 6821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:52.930927 6807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:52.931061 6807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c4218904dd7e42d2be95b901a6e5db92"
format_stamp: "Formatted at 2024-05-03 07:58:52 on dist-test-slave-bj8b"
I20240503 07:58:52.931164 6807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:52.958161 6807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:52.958923 6807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:52.959105 6807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:52.959712 6807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:52.960702 6807 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:52.960755 6807 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:52.960812 6807 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:52.960840 6807 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:52.969624 6807 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:40655
I20240503 07:58:52.969636 6934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:40655 every 8 connection(s)
I20240503 07:58:52.970763 6807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:58:52.978245 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 6807
I20240503 07:58:52.978467 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:58:52.980532 6935 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44703
I20240503 07:58:52.980796 6935 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:52.980778 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:52.981397 6935 heartbeater.cc:511] Master 127.0.99.190:44703 requested a full tablet report, sending...
I20240503 07:58:52.983117 6746 ts_manager.cc:194] Registered new tserver with Master: c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:58:52.984442 6746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:34745
W20240503 07:58:53.099824 6938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:53.100117 6938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:53.100167 6938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:53.103477 6938 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:53.103569 6938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:53.103657 6938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:58:53.107976 6938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--never_fsync=true
--heap_profile_path=/tmp/kudu.6938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:53.109078 6938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:53.110455 6938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:53.120200 6946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.120636 6943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.121234 6944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:53.121831 6938 server_base.cc:1034] running on GCE node
I20240503 07:58:53.122283 6938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:53.122982 6938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:53.124156 6938 hybrid_clock.cc:648] HybridClock initialized: now 1714723133124132 us; error 41 us; skew 500 ppm
I20240503 07:58:53.126436 6938 webserver.cc:460] Webserver started at http://127.0.99.130:46107/ using document root <none> and password file <none>
I20240503 07:58:53.127070 6938 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:53.127132 6938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:53.127359 6938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:53.129226 6938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9c085a3241f14cb7ba1332b508f4b57d"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.129750 6938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9c085a3241f14cb7ba1332b508f4b57d"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.133474 6938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.004s
I20240503 07:58:53.136094 6952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.137281 6938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:53.137410 6938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9c085a3241f14cb7ba1332b508f4b57d"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.137508 6938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:53.160915 6938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:53.161657 6938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:53.161837 6938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:53.162444 6938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:53.163455 6938 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:53.163514 6938 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.163571 6938 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:53.163595 6938 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.172291 7065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:46383 every 8 connection(s)
I20240503 07:58:53.172286 6938 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:46383
I20240503 07:58:53.173424 6938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:58:53.177436 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 6938
I20240503 07:58:53.177565 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:58:53.180361 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:53.184777 7066 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44703
I20240503 07:58:53.185119 7066 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:53.185840 7066 heartbeater.cc:511] Master 127.0.99.190:44703 requested a full tablet report, sending...
I20240503 07:58:53.187211 6746 ts_manager.cc:194] Registered new tserver with Master: 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:58:53.187891 6746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:45981
W20240503 07:58:53.298858 7069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:53.299136 7069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:53.299190 7069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:53.302479 7069 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:53.302567 7069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:53.302652 7069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:58:53.306682 7069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--never_fsync=true
--heap_profile_path=/tmp/kudu.7069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:53.307801 7069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:53.309070 7069 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:53.318643 7074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.319101 7077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.319584 7075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:53.320227 7069 server_base.cc:1034] running on GCE node
I20240503 07:58:53.320713 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:53.321424 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:53.322595 7069 hybrid_clock.cc:648] HybridClock initialized: now 1714723133322570 us; error 42 us; skew 500 ppm
I20240503 07:58:53.324765 7069 webserver.cc:460] Webserver started at http://127.0.99.131:45655/ using document root <none> and password file <none>
I20240503 07:58:53.325407 7069 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:53.325471 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:53.325676 7069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:53.327581 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "00a3f8e159c64327832d79fb32cdcca8"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.328081 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "00a3f8e159c64327832d79fb32cdcca8"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.331772 7069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:53.334475 7083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.335677 7069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:53.335803 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "00a3f8e159c64327832d79fb32cdcca8"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.335907 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:53.355994 7069 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:53.356774 7069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:53.356962 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:53.357594 7069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:53.358651 7069 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:53.358705 7069 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.358762 7069 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:53.358790 7069 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.367446 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:36317
I20240503 07:58:53.367470 7196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:36317 every 8 connection(s)
I20240503 07:58:53.368510 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:58:53.376786 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 7069
I20240503 07:58:53.376932 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:58:53.377847 7197 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44703
I20240503 07:58:53.378109 7197 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:53.378671 7197 heartbeater.cc:511] Master 127.0.99.190:44703 requested a full tablet report, sending...
I20240503 07:58:53.379148 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:53.379756 6746 ts_manager.cc:194] Registered new tserver with Master: 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:58:53.380507 6746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:54151
W20240503 07:58:53.497407 7200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:53.497732 7200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:53.497803 7200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:53.501242 7200 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:53.501351 7200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:53.501461 7200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:58:53.505600 7200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--never_fsync=true
--heap_profile_path=/tmp/kudu.7200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:53.506726 7200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:53.507988 7200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:53.517343 7205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.517798 7208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.518311 7206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:53.518921 7200 server_base.cc:1034] running on GCE node
I20240503 07:58:53.519361 7200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:53.520006 7200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:53.521162 7200 hybrid_clock.cc:648] HybridClock initialized: now 1714723133521140 us; error 35 us; skew 500 ppm
I20240503 07:58:53.523334 7200 webserver.cc:460] Webserver started at http://127.0.99.132:46745/ using document root <none> and password file <none>
I20240503 07:58:53.523943 7200 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:53.524005 7200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:53.524190 7200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:53.526047 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5227b26243484bbfb177a995ceb50afc"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.526594 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5227b26243484bbfb177a995ceb50afc"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.530263 7200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 07:58:53.532683 7214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.533905 7200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:53.534034 7200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5227b26243484bbfb177a995ceb50afc"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.534132 7200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:53.550257 7200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:53.551108 7200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:53.551301 7200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:53.551930 7200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:53.552975 7200 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:53.553030 7200 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.553086 7200 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:53.553115 7200 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.561877 7200 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:37483
I20240503 07:58:53.561892 7327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:37483 every 8 connection(s)
I20240503 07:58:53.562965 7200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:58:53.565760 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 7200
I20240503 07:58:53.565912 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:58:53.568619 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:53.573704 7328 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44703
I20240503 07:58:53.573985 7328 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:53.574617 7328 heartbeater.cc:511] Master 127.0.99.190:44703 requested a full tablet report, sending...
I20240503 07:58:53.575771 6746 ts_manager.cc:194] Registered new tserver with Master: 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:58:53.576437 6746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:38593
W20240503 07:58:53.687186 7331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:53.687469 7331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:53.687536 7331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:53.690881 7331 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:53.690973 7331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:53.691071 7331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:58:53.695070 7331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--never_fsync=true
--heap_profile_path=/tmp/kudu.7331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:53.696148 7331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:53.697484 7331 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:53.707140 7336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.707576 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.708117 7337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:53.708719 7331 server_base.cc:1034] running on GCE node
I20240503 07:58:53.709156 7331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:53.709846 7331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:53.711004 7331 hybrid_clock.cc:648] HybridClock initialized: now 1714723133710991 us; error 35 us; skew 500 ppm
I20240503 07:58:53.713281 7331 webserver.cc:460] Webserver started at http://127.0.99.133:42785/ using document root <none> and password file <none>
I20240503 07:58:53.713928 7331 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:53.714008 7331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:53.714264 7331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:53.716138 7331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "3befbb2d206f4be89d3e07f38837374b"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.716640 7331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "3befbb2d206f4be89d3e07f38837374b"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.720510 7331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 07:58:53.723022 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.724365 7331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:53.724498 7331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3befbb2d206f4be89d3e07f38837374b"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.724611 7331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:53.738593 7331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:53.739457 7331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:53.739635 7331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:53.740280 7331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:53.741353 7331 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:53.741405 7331 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.741452 7331 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:53.741483 7331 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.749904 7331 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:37815
I20240503 07:58:53.749960 7458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:37815 every 8 connection(s)
I20240503 07:58:53.751011 7331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:58:53.755568 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 7331
I20240503 07:58:53.755704 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:58:53.757972 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:53.761073 7459 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44703
I20240503 07:58:53.761363 7459 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:53.761883 7459 heartbeater.cc:511] Master 127.0.99.190:44703 requested a full tablet report, sending...
I20240503 07:58:53.762972 6746 ts_manager.cc:194] Registered new tserver with Master: 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:58:53.763545 6746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:55163
W20240503 07:58:53.872495 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:53.872788 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:53.872853 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:53.876206 7462 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:53.876302 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:53.876396 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:58:53.880450 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:53.881506 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:53.882786 7462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:53.892697 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.892985 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.893464 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:53.894122 7462 server_base.cc:1034] running on GCE node
I20240503 07:58:53.894637 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:53.895290 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:53.896457 7462 hybrid_clock.cc:648] HybridClock initialized: now 1714723133896438 us; error 39 us; skew 500 ppm
I20240503 07:58:53.898721 7462 webserver.cc:460] Webserver started at http://127.0.99.134:41199/ using document root <none> and password file <none>
I20240503 07:58:53.899386 7462 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:53.899454 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:53.899653 7462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:53.901593 7462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "cb73994515af4880b05383e78368bb4a"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.902112 7462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "cb73994515af4880b05383e78368bb4a"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.906069 7462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 07:58:53.908540 7476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.909711 7462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:53.909837 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "cb73994515af4880b05383e78368bb4a"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:58:53.909938 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:53.928462 7462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:53.929179 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:53.929360 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:53.929986 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:53.930996 7462 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:53.931048 7462 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.931102 7462 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:53.931125 7462 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.940285 7462 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:45435
I20240503 07:58:53.940310 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:45435 every 8 connection(s)
I20240503 07:58:53.941468 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:58:53.944953 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 7462
I20240503 07:58:53.945083 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 07:58:53.947283 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:53.951689 7590 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44703
I20240503 07:58:53.951964 7590 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:53.952502 7590 heartbeater.cc:511] Master 127.0.99.190:44703 requested a full tablet report, sending...
I20240503 07:58:53.953631 6746 ts_manager.cc:194] Registered new tserver with Master: cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:58:53.954257 6746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:37015
I20240503 07:58:53.987948 6935 heartbeater.cc:503] Master 127.0.99.190:44703 was elected leader, sending a full tablet report...
W20240503 07:58:54.065274 7593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:54.065565 7593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:54.065634 7593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:54.069016 7593 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:54.069123 7593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:54.069219 7593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:58:54.073376 7593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--never_fsync=true
--heap_profile_path=/tmp/kudu.7593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:54.074574 7593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:54.075924 7593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:54.085654 7601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.086161 7598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.086737 7599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:54.087334 7593 server_base.cc:1034] running on GCE node
I20240503 07:58:54.087848 7593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:54.088531 7593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:54.089720 7593 hybrid_clock.cc:648] HybridClock initialized: now 1714723134089689 us; error 48 us; skew 500 ppm
I20240503 07:58:54.092191 7593 webserver.cc:460] Webserver started at http://127.0.99.135:42821/ using document root <none> and password file <none>
I20240503 07:58:54.093024 7593 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:54.093107 7593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:54.093360 7593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:54.095420 7593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "159177b3caeb434baa98e04136d050b2"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-bj8b"
I20240503 07:58:54.095973 7593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "159177b3caeb434baa98e04136d050b2"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-bj8b"
I20240503 07:58:54.100157 7593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 07:58:54.102867 7607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.104120 7593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:54.104269 7593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "159177b3caeb434baa98e04136d050b2"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-bj8b"
I20240503 07:58:54.104373 7593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:54.129446 7593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:54.130158 7593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:54.130376 7593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:54.130971 7593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:54.131984 7593 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:54.132036 7593 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.132093 7593 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:54.132117 7593 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.140777 7593 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:43369
I20240503 07:58:54.140796 7720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:43369 every 8 connection(s)
I20240503 07:58:54.141830 7593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:58:54.143805 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 7593
I20240503 07:58:54.143937 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 07:58:54.146165 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:58:54.151239 7721 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44703
I20240503 07:58:54.151538 7721 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:54.152192 7721 heartbeater.cc:511] Master 127.0.99.190:44703 requested a full tablet report, sending...
I20240503 07:58:54.153343 6746 ts_manager.cc:194] Registered new tserver with Master: 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:58:54.153909 6746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:39565
I20240503 07:58:54.190752 7066 heartbeater.cc:503] Master 127.0.99.190:44703 was elected leader, sending a full tablet report...
W20240503 07:58:54.264642 7724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:54.264923 7724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:54.264972 7724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:54.268193 7724 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:58:54.268286 7724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:54.268359 7724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:58:54.272374 7724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44703
--never_fsync=true
--heap_profile_path=/tmp/kudu.7724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:54.273442 7724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:54.274772 7724 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:54.284359 7732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.284765 7730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.285346 7729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:54.285898 7724 server_base.cc:1034] running on GCE node
I20240503 07:58:54.286394 7724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:54.287055 7724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:54.288208 7724 hybrid_clock.cc:648] HybridClock initialized: now 1714723134288191 us; error 53 us; skew 500 ppm
I20240503 07:58:54.290454 7724 webserver.cc:460] Webserver started at http://127.0.99.136:43201/ using document root <none> and password file <none>
I20240503 07:58:54.291064 7724 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:54.291128 7724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:54.291319 7724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:54.293255 7724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "a372e98a670f49e9adca46ab5c836c4a"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-bj8b"
I20240503 07:58:54.293783 7724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "a372e98a670f49e9adca46ab5c836c4a"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-bj8b"
I20240503 07:58:54.297662 7724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 07:58:54.300271 7738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.301465 7724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:54.301596 7724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a372e98a670f49e9adca46ab5c836c4a"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-bj8b"
I20240503 07:58:54.301723 7724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:54.323963 7724 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:54.324800 7724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:54.324975 7724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:54.325601 7724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:54.326709 7724 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:54.326766 7724 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.326812 7724 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:54.326833 7724 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.336094 7724 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:43683
I20240503 07:58:54.336122 7851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:43683 every 8 connection(s)
I20240503 07:58:54.337195 7724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:58:54.342571 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 7724
I20240503 07:58:54.342707 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 07:58:54.347026 7852 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44703
I20240503 07:58:54.347280 7852 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:54.347816 7852 heartbeater.cc:511] Master 127.0.99.190:44703 requested a full tablet report, sending...
I20240503 07:58:54.348896 6746 ts_manager.cc:194] Registered new tserver with Master: a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:58:54.349488 6746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:40227
I20240503 07:58:54.357311 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 07:58:54.366976 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 07:58:54.367067 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 7331
I20240503 07:58:54.372406 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 7462
I20240503 07:58:54.377812 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 7593
I20240503 07:58:54.382704 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu with pid 7724
I20240503 07:58:54.382704 7197 heartbeater.cc:503] Master 127.0.99.190:44703 was elected leader, sending a full tablet report...
I20240503 07:58:54.578836 7328 heartbeater.cc:503] Master 127.0.99.190:44703 was elected leader, sending a full tablet report...
I20240503 07:58:58.139204 6746 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:58.160712 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 0a53d599d057465496a555820cd6742d (DEFAULT_TABLE table=rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.160573 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 24bfdc19078046259a38df8e61fdaee5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.161012 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 24bfdc19078046259a38df8e61fdaee5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.161746 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bfdc19078046259a38df8e61fdaee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.161495 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 0a53d599d057465496a555820cd6742d (DEFAULT_TABLE table=rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.161978 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bfdc19078046259a38df8e61fdaee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.162088 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a53d599d057465496a555820cd6742d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.162312 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a53d599d057465496a555820cd6742d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.162629 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 779675bcd7d7497ebe50cbb1875fb1e7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.162875 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779675bcd7d7497ebe50cbb1875fb1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.160573 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 779675bcd7d7497ebe50cbb1875fb1e7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.164508 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779675bcd7d7497ebe50cbb1875fb1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.169339 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 0a53d599d057465496a555820cd6742d (DEFAULT_TABLE table=rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.170101 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 779675bcd7d7497ebe50cbb1875fb1e7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.170397 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779675bcd7d7497ebe50cbb1875fb1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.170425 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a53d599d057465496a555820cd6742d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.171317 7873 tablet_bootstrap.cc:492] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:58:58.169337 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 24bfdc19078046259a38df8e61fdaee5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.173858 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bfdc19078046259a38df8e61fdaee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.174549 7873 tablet_bootstrap.cc:654] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.175393 7873 log.cc:824] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:58.176219 7874 tablet_bootstrap.cc:492] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:58:58.177814 7872 tablet_bootstrap.cc:492] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:58:58.179351 7874 tablet_bootstrap.cc:654] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.179970 7872 tablet_bootstrap.cc:654] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.180133 7874 log.cc:824] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:58.180656 7872 log.cc:824] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:58.180887 7876 tablet_bootstrap.cc:492] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:58:58.182330 7873 tablet_bootstrap.cc:492] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:58:58.182415 7874 tablet_bootstrap.cc:492] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:58:58.182415 7872 tablet_bootstrap.cc:492] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:58:58.182570 7873 ts_tablet_manager.cc:1367] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240503 07:58:58.182592 7872 ts_tablet_manager.cc:1367] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:58:58.182592 7874 ts_tablet_manager.cc:1367] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 07:58:58.185773 7873 raft_consensus.cc:348] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.185989 7873 raft_consensus.cc:374] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.185871 7872 raft_consensus.cc:348] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.186039 7873 raft_consensus.cc:729] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.186128 7872 raft_consensus.cc:374] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.186226 7872 raft_consensus.cc:729] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.186280 7874 raft_consensus.cc:348] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:58:58.186519 7874 raft_consensus.cc:374] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.186578 7874 raft_consensus.cc:729] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.186776 7872 consensus_queue.cc:260] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.186976 7874 consensus_queue.cc:260] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:58:58.187784 7874 ts_tablet_manager.cc:1398] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:58:58.187784 7872 ts_tablet_manager.cc:1398] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:58:58.188076 7872 tablet_bootstrap.cc:492] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:58:58.188144 7874 tablet_bootstrap.cc:492] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:58:58.188292 7876 tablet_bootstrap.cc:654] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.188658 7873 consensus_queue.cc:260] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.189066 7872 tablet_bootstrap.cc:654] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.189066 7874 tablet_bootstrap.cc:654] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.190120 7873 ts_tablet_manager.cc:1398] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20240503 07:58:58.191618 7873 tablet_bootstrap.cc:492] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:58:58.191617 7876 log.cc:824] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:58.192935 7873 tablet_bootstrap.cc:654] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.192955 7874 tablet_bootstrap.cc:492] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:58:58.193039 7874 ts_tablet_manager.cc:1367] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:58:58.193071 7872 tablet_bootstrap.cc:492] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:58:58.193141 7872 ts_tablet_manager.cc:1367] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:58:58.193627 7872 raft_consensus.cc:348] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.193629 7874 raft_consensus.cc:348] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.193737 7874 raft_consensus.cc:374] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.193737 7872 raft_consensus.cc:374] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.193768 7874 raft_consensus.cc:729] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.193768 7872 raft_consensus.cc:729] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.193909 7874 consensus_queue.cc:260] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.193920 7872 consensus_queue.cc:260] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.194216 7874 ts_tablet_manager.cc:1398] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:58.194216 7872 ts_tablet_manager.cc:1398] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:58.194399 7873 tablet_bootstrap.cc:492] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:58:58.194413 7872 tablet_bootstrap.cc:492] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:58:58.194489 7873 ts_tablet_manager.cc:1367] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:58.195046 7873 raft_consensus.cc:348] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:58:58.195171 7873 raft_consensus.cc:374] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.195201 7873 raft_consensus.cc:729] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.195387 7872 tablet_bootstrap.cc:654] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.195349 7873 consensus_queue.cc:260] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:58:58.195645 7873 ts_tablet_manager.cc:1398] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:58.196907 7872 tablet_bootstrap.cc:492] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:58:58.196990 7872 ts_tablet_manager.cc:1367] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:58.197129 7876 tablet_bootstrap.cc:492] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:58:58.197345 7876 ts_tablet_manager.cc:1367] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240503 07:58:58.197640 7872 raft_consensus.cc:348] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:58:58.197744 7872 raft_consensus.cc:374] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.197772 7872 raft_consensus.cc:729] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.197893 7872 consensus_queue.cc:260] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:58:58.198154 7872 ts_tablet_manager.cc:1398] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:58.201648 7876 raft_consensus.cc:348] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.201874 7876 raft_consensus.cc:374] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.201952 7876 raft_consensus.cc:729] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.202420 7876 consensus_queue.cc:260] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.203356 7876 ts_tablet_manager.cc:1398] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 07:58:58.203716 7876 tablet_bootstrap.cc:492] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:58:58.204675 7876 tablet_bootstrap.cc:654] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.205860 7876 tablet_bootstrap.cc:492] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:58:58.205943 7876 ts_tablet_manager.cc:1367] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:58.206504 7876 raft_consensus.cc:348] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.206607 7876 raft_consensus.cc:374] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.206631 7876 raft_consensus.cc:729] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.206753 7876 consensus_queue.cc:260] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.207034 7876 ts_tablet_manager.cc:1398] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:58.216665 7879 raft_consensus.cc:482] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:58.216902 7879 raft_consensus.cc:504] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:58:58.218137 7879 leader_election.cc:288] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:58:58.222049 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779675bcd7d7497ebe50cbb1875fb1e7" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:58:58.222049 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779675bcd7d7497ebe50cbb1875fb1e7" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:58:58.222395 7282 raft_consensus.cc:2454] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:58:58.222395 7020 raft_consensus.cc:2454] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:58:58.222854 6824 leader_election.cc:302] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:58:58.223120 7879 raft_consensus.cc:2781] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:58.223194 7879 raft_consensus.cc:482] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:58.223240 7879 raft_consensus.cc:3037] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.224221 7879 raft_consensus.cc:504] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:58:58.224575 7879 leader_election.cc:288] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:58:58.224961 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779675bcd7d7497ebe50cbb1875fb1e7" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:58:58.224994 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779675bcd7d7497ebe50cbb1875fb1e7" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:58:58.225090 7282 raft_consensus.cc:3037] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.225104 7020 raft_consensus.cc:3037] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.225996 7020 raft_consensus.cc:2454] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:58:58.226011 7282 raft_consensus.cc:2454] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
W20240503 07:58:58.226078 6936 tablet.cc:2369] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:58.226285 6824 leader_election.cc:302] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:58:58.226581 7879 raft_consensus.cc:2781] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:58.226827 7879 raft_consensus.cc:686] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:58:58.227167 7879 consensus_queue.cc:237] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:58:58.230051 6747 catalog_manager.cc:5484] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:58:58.313998 7879 raft_consensus.cc:482] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:58.314196 7879 raft_consensus.cc:504] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.314769 7879 leader_election.cc:288] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:58:58.315259 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bfdc19078046259a38df8e61fdaee5" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:58:58.315438 7020 raft_consensus.cc:2454] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:58:58.315755 6824 leader_election.cc:302] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:58:58.315930 7879 raft_consensus.cc:2781] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:58.315986 7879 raft_consensus.cc:482] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:58.316022 7879 raft_consensus.cc:3037] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.317152 7879 raft_consensus.cc:504] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.317607 7879 leader_election.cc:288] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
W20240503 07:58:58.317646 7329 tablet.cc:2369] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:58.317955 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bfdc19078046259a38df8e61fdaee5" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:58:58.318094 7020 raft_consensus.cc:3037] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.319044 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bfdc19078046259a38df8e61fdaee5" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:58:58.319051 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bfdc19078046259a38df8e61fdaee5" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:58:58.319185 7020 raft_consensus.cc:2454] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:58:58.319366 7151 raft_consensus.cc:2454] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:58:58.319528 6824 leader_election.cc:302] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:58:58.319717 7879 raft_consensus.cc:2781] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:58.319813 7879 raft_consensus.cc:686] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:58:58.319972 7879 consensus_queue.cc:237] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.321779 6747 catalog_manager.cc:5484] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
W20240503 07:58:58.373675 7198 tablet.cc:2369] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:58:58.428747 7067 tablet.cc:2369] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:58.429145 7879 raft_consensus.cc:482] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:58.429291 7879 raft_consensus.cc:504] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.429764 7879 leader_election.cc:288] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:58:58.430337 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a53d599d057465496a555820cd6742d" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:58:58.430403 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a53d599d057465496a555820cd6742d" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:58:58.430516 7282 raft_consensus.cc:2454] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:58:58.430568 7150 raft_consensus.cc:2454] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:58:58.430884 6825 leader_election.cc:302] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:58:58.431066 7879 raft_consensus.cc:2781] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:58.431126 7879 raft_consensus.cc:482] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:58.431151 7879 raft_consensus.cc:3037] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.432000 7879 raft_consensus.cc:504] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.432372 7879 leader_election.cc:288] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:58:58.432816 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a53d599d057465496a555820cd6742d" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:58:58.432837 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a53d599d057465496a555820cd6742d" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:58:58.432984 7282 raft_consensus.cc:3037] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.432996 7150 raft_consensus.cc:3037] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.433826 7282 raft_consensus.cc:2454] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:58:58.433883 7150 raft_consensus.cc:2454] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:58:58.434185 6825 leader_election.cc:302] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:58:58.434374 7879 raft_consensus.cc:2781] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:58.434468 7879 raft_consensus.cc:686] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:58:58.434612 7879 consensus_queue.cc:237] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.436408 6747 catalog_manager.cc:5484] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
W20240503 07:58:58.571156 7895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:58.571460 7895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:58.574839 7895 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 07:58:58.670025 7150 raft_consensus.cc:1261] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:58.670087 7282 raft_consensus.cc:1261] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:58.673261 7020 raft_consensus.cc:1261] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:58.673610 7019 raft_consensus.cc:1261] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:58.674270 7891 consensus_queue.cc:1035] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:58.674500 7282 raft_consensus.cc:1261] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:58.675161 7924 consensus_queue.cc:1035] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:58.675238 7891 consensus_queue.cc:1035] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:58.677448 7150 raft_consensus.cc:3037] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.678835 7150 raft_consensus.cc:1261] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:58.680758 7879 consensus_queue.cc:1035] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:58.672446 7923 consensus_queue.cc:1035] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:58.686971 7925 consensus_queue.cc:1035] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:58.688597 7921 mvcc.cc:205] Tried to move back new op lower bound from 7023505975996297216 to 7023505974560653312. Current Snapshot: MvccSnapshot[applied={T|T < 7023505975982698496}]
I20240503 07:58:58.696208 7926 mvcc.cc:205] Tried to move back new op lower bound from 7023505975996297216 to 7023505974560653312. Current Snapshot: MvccSnapshot[applied={T|T < 7023505975982698496}]
I20240503 07:58:58.712394 7941 mvcc.cc:205] Tried to move back new op lower bound from 7023505975998103552 to 7023505975029719040. Current Snapshot: MvccSnapshot[applied={T|T < 7023505975982977024}]
I20240503 07:58:58.723109 7937 mvcc.cc:205] Tried to move back new op lower bound from 7023505975996297216 to 7023505974560653312. Current Snapshot: MvccSnapshot[applied={T|T < 7023505975982698496}]
INSERT report
rows total: 2000
time total: 122.149 ms
time per row: 0.0610743 ms
I20240503 07:58:58.752454 6743 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:58.758862 7131 tablet_service.cc:1467] Processing CreateTablet for tablet ea625a1834fb4a038a32a90d2fcbf1ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=7240d8c6aeff4067b4b95979e77f21da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.758971 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 660355a4482d4a30bbb71cd0b56527d1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7240d8c6aeff4067b4b95979e77f21da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.759227 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660355a4482d4a30bbb71cd0b56527d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.759238 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea625a1834fb4a038a32a90d2fcbf1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.759356 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 5293a26b65484ae5a9e76469e644c5bc (DEFAULT_TABLE table=rebalance_test_table_1 [id=7240d8c6aeff4067b4b95979e77f21da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.759583 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5293a26b65484ae5a9e76469e644c5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.761318 7952 tablet_bootstrap.cc:492] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:58:58.761547 7953 tablet_bootstrap.cc:492] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:58:58.761687 6999 tablet_service.cc:1467] Processing CreateTablet for tablet ea625a1834fb4a038a32a90d2fcbf1ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=7240d8c6aeff4067b4b95979e77f21da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.761912 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea625a1834fb4a038a32a90d2fcbf1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.762208 7952 tablet_bootstrap.cc:654] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.762499 7953 tablet_bootstrap.cc:654] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.763240 7952 tablet_bootstrap.cc:492] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:58:58.763321 7952 ts_tablet_manager.cc:1367] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:58.763646 7953 tablet_bootstrap.cc:492] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:58:58.763724 7953 ts_tablet_manager.cc:1367] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:58.763700 7952 raft_consensus.cc:348] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:58.763782 7952 raft_consensus.cc:374] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.763811 7952 raft_consensus.cc:729] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.763914 7952 consensus_queue.cc:260] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:58.764196 7952 ts_tablet_manager.cc:1398] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:58.764178 7953 raft_consensus.cc:348] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:58.764278 7953 raft_consensus.cc:374] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.764308 7953 raft_consensus.cc:729] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.764359 7952 tablet_bootstrap.cc:492] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:58:58.764401 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 5293a26b65484ae5a9e76469e644c5bc (DEFAULT_TABLE table=rebalance_test_table_1 [id=7240d8c6aeff4067b4b95979e77f21da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.764457 7953 consensus_queue.cc:260] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:58.764691 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5293a26b65484ae5a9e76469e644c5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.765211 7952 tablet_bootstrap.cc:654] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.766032 7953 ts_tablet_manager.cc:1398] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:58.767112 7262 tablet_service.cc:1467] Processing CreateTablet for tablet ea625a1834fb4a038a32a90d2fcbf1ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=7240d8c6aeff4067b4b95979e77f21da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.767413 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea625a1834fb4a038a32a90d2fcbf1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.767587 7957 tablet_bootstrap.cc:492] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:58:58.768160 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 5293a26b65484ae5a9e76469e644c5bc (DEFAULT_TABLE table=rebalance_test_table_1 [id=7240d8c6aeff4067b4b95979e77f21da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.768234 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 660355a4482d4a30bbb71cd0b56527d1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7240d8c6aeff4067b4b95979e77f21da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.768404 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5293a26b65484ae5a9e76469e644c5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.768471 7957 tablet_bootstrap.cc:654] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.768697 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660355a4482d4a30bbb71cd0b56527d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.770033 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 660355a4482d4a30bbb71cd0b56527d1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7240d8c6aeff4067b4b95979e77f21da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.770237 7957 tablet_bootstrap.cc:492] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:58:58.770282 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660355a4482d4a30bbb71cd0b56527d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.770329 7957 ts_tablet_manager.cc:1367] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:58:58.770886 7953 tablet_bootstrap.cc:492] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:58:58.770859 7957 raft_consensus.cc:348] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:58.770973 7957 raft_consensus.cc:374] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.771000 7957 raft_consensus.cc:729] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.771131 7957 consensus_queue.cc:260] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:58.771464 7957 ts_tablet_manager.cc:1398] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:58.771836 7953 tablet_bootstrap.cc:654] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.773173 7953 tablet_bootstrap.cc:492] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:58:58.773279 7953 ts_tablet_manager.cc:1367] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:58.773653 7953 raft_consensus.cc:348] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:58.773733 7953 raft_consensus.cc:374] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.773753 7953 raft_consensus.cc:729] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.773881 7953 consensus_queue.cc:260] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:58.774093 7952 tablet_bootstrap.cc:492] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:58:58.774116 7953 ts_tablet_manager.cc:1398] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:58.774223 7952 ts_tablet_manager.cc:1367] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 07:58:58.774276 7953 tablet_bootstrap.cc:492] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:58:58.774823 7952 raft_consensus.cc:348] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.774937 7952 raft_consensus.cc:374] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.774969 7952 raft_consensus.cc:729] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.775192 7953 tablet_bootstrap.cc:654] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.775136 7952 consensus_queue.cc:260] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.775566 7952 ts_tablet_manager.cc:1398] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:58:58.775733 7952 tablet_bootstrap.cc:492] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:58:58.776320 7953 tablet_bootstrap.cc:492] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:58:58.776405 7953 ts_tablet_manager.cc:1367] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:58.776695 7952 tablet_bootstrap.cc:654] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.776852 7953 raft_consensus.cc:348] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.776952 7953 raft_consensus.cc:374] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.776983 7953 raft_consensus.cc:729] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.777114 7953 consensus_queue.cc:260] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.777403 7953 ts_tablet_manager.cc:1398] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:58.777457 7959 tablet_bootstrap.cc:492] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:58:58.778538 7959 tablet_bootstrap.cc:654] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.778935 7952 tablet_bootstrap.cc:492] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:58:58.779032 7952 ts_tablet_manager.cc:1367] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:58.779556 7952 raft_consensus.cc:348] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:58.779687 7952 raft_consensus.cc:374] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.779734 7952 raft_consensus.cc:729] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.780159 7952 consensus_queue.cc:260] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:58.780423 7952 ts_tablet_manager.cc:1398] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:58.780567 7959 tablet_bootstrap.cc:492] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:58:58.780684 7959 ts_tablet_manager.cc:1367] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:58.781148 7959 raft_consensus.cc:348] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:58.781258 7959 raft_consensus.cc:374] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.781287 7959 raft_consensus.cc:729] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.781426 7959 consensus_queue.cc:260] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:58.781673 7959 ts_tablet_manager.cc:1398] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:58.781834 7959 tablet_bootstrap.cc:492] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:58:58.782922 7959 tablet_bootstrap.cc:654] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.784214 7959 tablet_bootstrap.cc:492] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:58:58.784299 7959 ts_tablet_manager.cc:1367] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:58.784696 7959 raft_consensus.cc:348] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.784794 7959 raft_consensus.cc:374] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.784823 7959 raft_consensus.cc:729] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.784945 7959 consensus_queue.cc:260] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.785198 7959 ts_tablet_manager.cc:1398] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:58.882867 7887 raft_consensus.cc:482] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:58.883044 7887 raft_consensus.cc:504] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.884337 7887 leader_election.cc:288] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:58:58.888712 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660355a4482d4a30bbb71cd0b56527d1" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:58:58.888922 7279 raft_consensus.cc:2454] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:58:58.889184 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660355a4482d4a30bbb71cd0b56527d1" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:58:58.889385 7020 raft_consensus.cc:2454] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:58:58.889339 7087 leader_election.cc:302] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:58:58.889640 7887 raft_consensus.cc:2781] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:58.889714 7887 raft_consensus.cc:482] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:58.889751 7887 raft_consensus.cc:3037] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.890690 7887 raft_consensus.cc:504] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.891069 7887 leader_election.cc:288] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:58:58.891505 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660355a4482d4a30bbb71cd0b56527d1" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:58:58.891633 7020 raft_consensus.cc:3037] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.892481 7020 raft_consensus.cc:2454] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:58:58.892764 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660355a4482d4a30bbb71cd0b56527d1" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:58:58.892800 7086 leader_election.cc:302] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:58:58.892915 7279 raft_consensus.cc:3037] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.892983 7887 raft_consensus.cc:2781] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:58.893201 7887 raft_consensus.cc:686] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:58:58.893549 7887 consensus_queue.cc:237] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:58:58.894006 7279 raft_consensus.cc:2454] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:58:58.895090 7881 raft_consensus.cc:482] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:58.895218 7881 raft_consensus.cc:504] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:58.896129 6746 catalog_manager.cc:5484] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:58:58.896490 7881 leader_election.cc:288] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:58:58.900688 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea625a1834fb4a038a32a90d2fcbf1ac" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:58:58.900842 7279 raft_consensus.cc:2454] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:58:58.900826 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea625a1834fb4a038a32a90d2fcbf1ac" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:58:58.900971 7151 raft_consensus.cc:2454] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:58:58.901221 6956 leader_election.cc:302] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:58:58.901515 7881 raft_consensus.cc:2781] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:58.901576 7881 raft_consensus.cc:482] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:58.901602 7881 raft_consensus.cc:3037] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.902500 7881 raft_consensus.cc:504] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:58.902855 7881 leader_election.cc:288] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:58:58.903223 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea625a1834fb4a038a32a90d2fcbf1ac" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:58:58.903302 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea625a1834fb4a038a32a90d2fcbf1ac" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:58:58.903348 7151 raft_consensus.cc:3037] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.903435 7279 raft_consensus.cc:3037] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.904150 7151 raft_consensus.cc:2454] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:58:58.904176 7279 raft_consensus.cc:2454] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:58:58.904474 6956 leader_election.cc:302] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:58:58.904652 7881 raft_consensus.cc:2781] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:58.904894 7881 raft_consensus.cc:686] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:58:58.905246 7881 consensus_queue.cc:237] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:58.907466 6746 catalog_manager.cc:5484] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:58:59.018436 7923 raft_consensus.cc:482] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:59.018613 7923 raft_consensus.cc:504] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:59.019109 7923 leader_election.cc:288] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:58:59.019641 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5293a26b65484ae5a9e76469e644c5bc" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:58:59.019641 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5293a26b65484ae5a9e76469e644c5bc" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:58:59.019814 7151 raft_consensus.cc:2454] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:58:59.019814 7020 raft_consensus.cc:2454] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:58:59.020179 6825 leader_election.cc:302] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:58:59.020483 7923 raft_consensus.cc:2781] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:59.020560 7923 raft_consensus.cc:482] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:59.020591 7923 raft_consensus.cc:3037] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.021456 7923 raft_consensus.cc:504] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:59.021822 7923 leader_election.cc:288] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:58:59.022238 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5293a26b65484ae5a9e76469e644c5bc" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:58:59.022258 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5293a26b65484ae5a9e76469e644c5bc" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:58:59.022404 7020 raft_consensus.cc:3037] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.022404 7151 raft_consensus.cc:3037] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.023380 7151 raft_consensus.cc:2454] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:58:59.023380 7020 raft_consensus.cc:2454] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:58:59.023741 6825 leader_election.cc:302] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:58:59.023916 7923 raft_consensus.cc:2781] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:59.024009 7923 raft_consensus.cc:686] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:58:59.024192 7923 consensus_queue.cc:237] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:59.026021 6746 catalog_manager.cc:5484] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
W20240503 07:58:59.188789 7977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:59.189112 7977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:59.192586 7977 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 07:58:59.280854 7020 raft_consensus.cc:1261] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.281246 7019 raft_consensus.cc:1261] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.281354 7279 raft_consensus.cc:1261] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.281754 7969 consensus_queue.cc:1035] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.282222 7969 consensus_queue.cc:1035] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.284852 7282 raft_consensus.cc:1261] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.285077 7151 raft_consensus.cc:1261] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.285238 7150 raft_consensus.cc:1261] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.285530 7925 consensus_queue.cc:1035] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.285908 7925 consensus_queue.cc:1035] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.287035 7881 consensus_queue.cc:1035] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.288718 7974 consensus_queue.cc:1035] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.412 ms
time per row: 0.0557062 ms
I20240503 07:58:59.350644 6746 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:59.356750 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 20deb333e99e43bb86262ed3390d29a0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.356802 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 20deb333e99e43bb86262ed3390d29a0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.357121 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20deb333e99e43bb86262ed3390d29a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.357127 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20deb333e99e43bb86262ed3390d29a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.357661 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8fccecada34616bdbf83d1f3ec9689 (DEFAULT_TABLE table=rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.357904 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8fccecada34616bdbf83d1f3ec9689. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.359154 8040 tablet_bootstrap.cc:492] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:58:59.359678 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 20deb333e99e43bb86262ed3390d29a0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.359962 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20deb333e99e43bb86262ed3390d29a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.359936 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 0123e273efc34f538c88705829865dfd (DEFAULT_TABLE table=rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.359927 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8fccecada34616bdbf83d1f3ec9689 (DEFAULT_TABLE table=rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.360172 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0123e273efc34f538c88705829865dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.360172 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8fccecada34616bdbf83d1f3ec9689. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.360211 8040 tablet_bootstrap.cc:654] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.361260 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8fccecada34616bdbf83d1f3ec9689 (DEFAULT_TABLE table=rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.361518 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8fccecada34616bdbf83d1f3ec9689. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.362260 8041 tablet_bootstrap.cc:492] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:58:59.362488 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 0123e273efc34f538c88705829865dfd (DEFAULT_TABLE table=rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.362766 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0123e273efc34f538c88705829865dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.363186 8041 tablet_bootstrap.cc:654] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.364122 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 0123e273efc34f538c88705829865dfd (DEFAULT_TABLE table=rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.364351 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0123e273efc34f538c88705829865dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.364398 8041 tablet_bootstrap.cc:492] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:58:59.364495 8041 ts_tablet_manager.cc:1367] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:59.364984 8045 tablet_bootstrap.cc:492] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:58:59.364979 8041 raft_consensus.cc:348] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:59.365121 8041 raft_consensus.cc:374] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.365173 8041 raft_consensus.cc:729] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.365347 8041 consensus_queue.cc:260] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:59.365778 8041 ts_tablet_manager.cc:1398] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:59.365971 8041 tablet_bootstrap.cc:492] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:58:59.365962 8043 tablet_bootstrap.cc:492] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:58:59.366917 8041 tablet_bootstrap.cc:654] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.367148 8040 tablet_bootstrap.cc:492] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:58:59.367338 8045 tablet_bootstrap.cc:654] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.367332 8040 ts_tablet_manager.cc:1367] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:58:59.368692 8043 tablet_bootstrap.cc:654] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.369652 8045 tablet_bootstrap.cc:492] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:58:59.369752 8045 ts_tablet_manager.cc:1367] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:58:59.369805 8043 tablet_bootstrap.cc:492] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:58:59.369925 8043 ts_tablet_manager.cc:1367] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:58:59.370198 8045 raft_consensus.cc:348] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.370306 8045 raft_consensus.cc:374] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.370333 8045 raft_consensus.cc:729] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.370322 8043 raft_consensus.cc:348] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.370473 8043 raft_consensus.cc:374] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.370514 8043 raft_consensus.cc:729] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.370561 8045 consensus_queue.cc:260] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.370651 8043 consensus_queue.cc:260] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.370884 8045 ts_tablet_manager.cc:1398] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:59.371002 8043 ts_tablet_manager.cc:1398] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:59.371210 8043 tablet_bootstrap.cc:492] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:58:59.372066 8043 tablet_bootstrap.cc:654] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.373255 8041 tablet_bootstrap.cc:492] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:58:59.373303 8043 tablet_bootstrap.cc:492] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:58:59.373360 8043 ts_tablet_manager.cc:1367] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:58:59.373360 8041 ts_tablet_manager.cc:1367] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:58:59.373788 8043 raft_consensus.cc:348] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:59.373790 8041 raft_consensus.cc:348] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.373890 8041 raft_consensus.cc:374] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.373890 8043 raft_consensus.cc:374] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.373920 8041 raft_consensus.cc:729] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.373920 8043 raft_consensus.cc:729] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.374034 8043 consensus_queue.cc:260] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:59.374040 8041 consensus_queue.cc:260] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.374323 8043 ts_tablet_manager.cc:1398] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:58:59.374323 8041 ts_tablet_manager.cc:1398] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:59.374521 8041 tablet_bootstrap.cc:492] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:58:59.375516 8041 tablet_bootstrap.cc:654] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.376953 8041 tablet_bootstrap.cc:492] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:58:59.377032 8041 ts_tablet_manager.cc:1367] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:59.377475 8041 raft_consensus.cc:348] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.377585 8041 raft_consensus.cc:374] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.377616 8041 raft_consensus.cc:729] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.367784 8040 raft_consensus.cc:348] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:59.377715 8040 raft_consensus.cc:374] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.377732 8041 consensus_queue.cc:260] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.377784 8040 raft_consensus.cc:729] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.378003 8041 ts_tablet_manager.cc:1398] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:59.377950 8040 consensus_queue.cc:260] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:59.379132 8040 ts_tablet_manager.cc:1398] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240503 07:58:59.379376 8040 tablet_bootstrap.cc:492] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:58:59.380285 8040 tablet_bootstrap.cc:654] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.383100 8040 tablet_bootstrap.cc:492] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:58:59.383211 8040 ts_tablet_manager.cc:1367] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:58:59.383669 8040 raft_consensus.cc:348] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.383805 8040 raft_consensus.cc:374] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.383841 8040 raft_consensus.cc:729] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.383968 8040 consensus_queue.cc:260] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.384245 8040 ts_tablet_manager.cc:1398] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:59.384416 8040 tablet_bootstrap.cc:492] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:58:59.385937 8040 tablet_bootstrap.cc:654] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.387193 8040 tablet_bootstrap.cc:492] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:58:59.387336 8040 ts_tablet_manager.cc:1367] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:59.387804 8040 raft_consensus.cc:348] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.387923 8040 raft_consensus.cc:374] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.387948 8040 raft_consensus.cc:729] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.388092 8040 consensus_queue.cc:260] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.388371 8040 ts_tablet_manager.cc:1398] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:59.436916 7969 raft_consensus.cc:482] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:59.437090 7969 raft_consensus.cc:504] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.437664 7969 leader_election.cc:288] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:58:59.438230 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0123e273efc34f538c88705829865dfd" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:58:59.438434 7282 raft_consensus.cc:2454] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:58:59.438815 7087 leader_election.cc:302] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:58:59.439028 7969 raft_consensus.cc:2781] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:59.439090 7969 raft_consensus.cc:482] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:59.439116 7969 raft_consensus.cc:3037] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.440299 7969 raft_consensus.cc:504] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.440706 7969 leader_election.cc:288] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:58:59.441224 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0123e273efc34f538c88705829865dfd" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:58:59.441377 7282 raft_consensus.cc:3037] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.442171 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0123e273efc34f538c88705829865dfd" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:58:59.442184 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0123e273efc34f538c88705829865dfd" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:58:59.442401 6888 raft_consensus.cc:3037] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.442638 7282 raft_consensus.cc:2454] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:58:59.443086 7087 leader_election.cc:302] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:58:59.443262 7969 raft_consensus.cc:2781] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:59.443336 6888 raft_consensus.cc:2454] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:58:59.443357 7969 raft_consensus.cc:686] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:58:59.443507 7969 consensus_queue.cc:237] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.445400 6747 catalog_manager.cc:5484] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:58:59.477229 8032 raft_consensus.cc:482] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:59.477383 8032 raft_consensus.cc:504] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.477958 8032 leader_election.cc:288] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:58:59.478601 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8fccecada34616bdbf83d1f3ec9689" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:58:59.478775 7282 raft_consensus.cc:2454] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:58:59.479143 6956 leader_election.cc:302] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:58:59.479346 8032 raft_consensus.cc:2781] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:59.479411 8032 raft_consensus.cc:482] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:59.479451 8032 raft_consensus.cc:3037] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.480295 8032 raft_consensus.cc:504] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.480677 8032 leader_election.cc:288] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:58:59.481139 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8fccecada34616bdbf83d1f3ec9689" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:58:59.481287 7282 raft_consensus.cc:3037] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.482131 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8fccecada34616bdbf83d1f3ec9689" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:58:59.482132 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8fccecada34616bdbf83d1f3ec9689" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:58:59.482276 7282 raft_consensus.cc:2454] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:58:59.482318 6888 raft_consensus.cc:2454] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:58:59.482585 6956 leader_election.cc:302] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:58:59.482762 8032 raft_consensus.cc:2781] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:59.482878 8032 raft_consensus.cc:686] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:58:59.483031 8032 consensus_queue.cc:237] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:58:59.484792 6747 catalog_manager.cc:5484] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:58:59.691233 8032 raft_consensus.cc:482] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:59.691407 8032 raft_consensus.cc:504] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:59.691888 8032 leader_election.cc:288] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:58:59.692431 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20deb333e99e43bb86262ed3390d29a0" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:58:59.692461 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20deb333e99e43bb86262ed3390d29a0" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:58:59.692584 6888 raft_consensus.cc:2454] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:58:59.692608 7282 raft_consensus.cc:2454] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:58:59.692934 6953 leader_election.cc:302] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:58:59.693140 8032 raft_consensus.cc:2781] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:59.693200 8032 raft_consensus.cc:482] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:59.693235 8032 raft_consensus.cc:3037] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.694141 8032 raft_consensus.cc:504] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:59.694562 8032 leader_election.cc:288] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:58:59.694972 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20deb333e99e43bb86262ed3390d29a0" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:58:59.695026 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20deb333e99e43bb86262ed3390d29a0" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:58:59.695113 7282 raft_consensus.cc:3037] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.695168 6888 raft_consensus.cc:3037] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.695947 7282 raft_consensus.cc:2454] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:58:59.696007 6888 raft_consensus.cc:2454] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:58:59.696256 6956 leader_election.cc:302] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:58:59.696432 8032 raft_consensus.cc:2781] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:59.696521 8032 raft_consensus.cc:686] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:58:59.696672 8032 consensus_queue.cc:237] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:58:59.698539 6747 catalog_manager.cc:5484] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
W20240503 07:58:59.853863 8058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:59.854249 8058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:59.858093 8058 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 07:58:59.899747 7887 consensus_queue.cc:1035] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.903275 7969 consensus_queue.cc:1035] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.953107 6888 raft_consensus.cc:3037] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.954273 6888 raft_consensus.cc:1261] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.955183 7279 raft_consensus.cc:1261] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.955977 8032 consensus_queue.cc:1035] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.957029 7974 consensus_queue.cc:1035] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.959239 7279 raft_consensus.cc:1261] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.960856 7974 consensus_queue.cc:1035] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.962711 6888 raft_consensus.cc:1261] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.963150 7974 consensus_queue.cc:1035] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 108.532 ms
time per row: 0.0542658 ms
I20240503 07:59:00.017298 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:00.023628 7131 tablet_service.cc:1467] Processing CreateTablet for tablet c27273ae45c449018143cc55e65336a6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.023670 6869 tablet_service.cc:1467] Processing CreateTablet for tablet c27273ae45c449018143cc55e65336a6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.023841 6999 tablet_service.cc:1467] Processing CreateTablet for tablet c27273ae45c449018143cc55e65336a6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.023990 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27273ae45c449018143cc55e65336a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.023994 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27273ae45c449018143cc55e65336a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.024184 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27273ae45c449018143cc55e65336a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.024670 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 5b794508aa77492398e48d25edfb8aa3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.024930 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b794508aa77492398e48d25edfb8aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.026731 7262 tablet_service.cc:1467] Processing CreateTablet for tablet abb55c09be324f37a1a546e73008a80d (DEFAULT_TABLE table=rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.026988 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abb55c09be324f37a1a546e73008a80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.026901 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 5b794508aa77492398e48d25edfb8aa3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.026901 7130 tablet_service.cc:1467] Processing CreateTablet for tablet abb55c09be324f37a1a546e73008a80d (DEFAULT_TABLE table=rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.027190 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b794508aa77492398e48d25edfb8aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.027495 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abb55c09be324f37a1a546e73008a80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.027829 8120 tablet_bootstrap.cc:492] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:00.028718 8120 tablet_bootstrap.cc:654] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.029297 8118 tablet_bootstrap.cc:492] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:00.029772 8120 tablet_bootstrap.cc:492] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:00.029862 8120 ts_tablet_manager.cc:1367] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:00.029863 6999 tablet_service.cc:1467] Processing CreateTablet for tablet abb55c09be324f37a1a546e73008a80d (DEFAULT_TABLE table=rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.030143 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abb55c09be324f37a1a546e73008a80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.030192 8118 tablet_bootstrap.cc:654] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.030310 8120 raft_consensus.cc:348] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.030458 8120 raft_consensus.cc:374] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.030489 8120 raft_consensus.cc:729] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.030654 8120 consensus_queue.cc:260] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.031049 8120 ts_tablet_manager.cc:1398] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.031225 8120 tablet_bootstrap.cc:492] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:00.031544 8117 tablet_bootstrap.cc:492] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:00.032081 8120 tablet_bootstrap.cc:654] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.032472 8117 tablet_bootstrap.cc:654] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.033196 8120 tablet_bootstrap.cc:492] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:00.033219 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 5b794508aa77492398e48d25edfb8aa3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.033285 8120 ts_tablet_manager.cc:1367] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:00.033475 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b794508aa77492398e48d25edfb8aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.033520 8119 tablet_bootstrap.cc:492] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:00.033758 8120 raft_consensus.cc:348] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:00.033857 8120 raft_consensus.cc:374] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.033883 8120 raft_consensus.cc:729] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.034001 8120 consensus_queue.cc:260] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:00.034292 8120 ts_tablet_manager.cc:1398] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.035305 8119 tablet_bootstrap.cc:654] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.035773 8118 tablet_bootstrap.cc:492] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:00.036492 8119 tablet_bootstrap.cc:492] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:00.036582 8119 ts_tablet_manager.cc:1367] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:00.036796 8117 tablet_bootstrap.cc:492] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:00.036898 8117 ts_tablet_manager.cc:1367] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:00.037081 8119 raft_consensus.cc:348] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.037223 8119 raft_consensus.cc:374] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.037257 8119 raft_consensus.cc:729] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.037360 8117 raft_consensus.cc:348] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.037400 8119 consensus_queue.cc:260] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.037459 8117 raft_consensus.cc:374] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.037493 8117 raft_consensus.cc:729] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.037653 8118 ts_tablet_manager.cc:1367] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:00.037680 8119 ts_tablet_manager.cc:1398] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.037876 8119 tablet_bootstrap.cc:492] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:00.038179 8118 raft_consensus.cc:348] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.038312 8118 raft_consensus.cc:374] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.038386 8118 raft_consensus.cc:729] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.038532 8118 consensus_queue.cc:260] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.038719 8117 consensus_queue.cc:260] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.038875 8119 tablet_bootstrap.cc:654] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.040014 8118 ts_tablet_manager.cc:1398] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:00.040210 8118 tablet_bootstrap.cc:492] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:00.040297 8117 ts_tablet_manager.cc:1398] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:00.040462 8117 tablet_bootstrap.cc:492] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:00.041565 8117 tablet_bootstrap.cc:654] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.041996 8118 tablet_bootstrap.cc:654] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.042881 8119 tablet_bootstrap.cc:492] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:00.042937 8117 tablet_bootstrap.cc:492] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:00.042974 8119 ts_tablet_manager.cc:1367] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:00.043023 8117 ts_tablet_manager.cc:1367] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:00.043146 8118 tablet_bootstrap.cc:492] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:00.043218 8118 ts_tablet_manager.cc:1367] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:00.043457 8119 raft_consensus.cc:348] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:00.043569 8119 raft_consensus.cc:374] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.043504 8117 raft_consensus.cc:348] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.043603 8119 raft_consensus.cc:729] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.043612 8117 raft_consensus.cc:374] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.043645 8117 raft_consensus.cc:729] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.043645 8118 raft_consensus.cc:348] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.043735 8118 raft_consensus.cc:374] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.043763 8118 raft_consensus.cc:729] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.043733 8119 consensus_queue.cc:260] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:00.043798 8117 consensus_queue.cc:260] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.043872 8118 consensus_queue.cc:260] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.044029 8119 ts_tablet_manager.cc:1398] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.044098 8117 ts_tablet_manager.cc:1398] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.044116 8118 ts_tablet_manager.cc:1398] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:00.044267 8118 tablet_bootstrap.cc:492] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:00.045186 8118 tablet_bootstrap.cc:654] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.046633 8118 tablet_bootstrap.cc:492] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:00.046717 8118 ts_tablet_manager.cc:1367] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:00.047101 8118 raft_consensus.cc:348] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:00.047230 8118 raft_consensus.cc:374] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.047260 8118 raft_consensus.cc:729] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.047384 8118 consensus_queue.cc:260] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:00.047653 8118 ts_tablet_manager.cc:1398] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.068053 7969 raft_consensus.cc:482] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.068181 7969 raft_consensus.cc:504] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.068619 7969 leader_election.cc:288] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:00.069036 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27273ae45c449018143cc55e65336a6" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:00.069069 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27273ae45c449018143cc55e65336a6" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:00.069187 7020 raft_consensus.cc:2454] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:00.069183 6888 raft_consensus.cc:2454] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:00.069551 7084 leader_election.cc:302] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:00.069729 7969 raft_consensus.cc:2781] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:00.069799 7969 raft_consensus.cc:482] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:00.069836 7969 raft_consensus.cc:3037] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.070755 7969 raft_consensus.cc:504] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.071087 7969 leader_election.cc:288] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:00.071447 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27273ae45c449018143cc55e65336a6" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:00.071491 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27273ae45c449018143cc55e65336a6" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:00.071575 6888 raft_consensus.cc:3037] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.071616 7020 raft_consensus.cc:3037] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.072425 7020 raft_consensus.cc:2454] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:00.072471 6888 raft_consensus.cc:2454] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:00.072746 7086 leader_election.cc:302] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:00.072939 7969 raft_consensus.cc:2781] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:00.073082 7969 raft_consensus.cc:686] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:00.073231 7969 consensus_queue.cc:237] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.075012 6738 catalog_manager.cc:5484] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:00.113780 7974 raft_consensus.cc:482] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.113926 7974 raft_consensus.cc:504] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:00.114459 7974 leader_election.cc:288] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:00.114840 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abb55c09be324f37a1a546e73008a80d" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:00.114899 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abb55c09be324f37a1a546e73008a80d" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:00.114991 7151 raft_consensus.cc:2454] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:00.115051 7279 raft_consensus.cc:2454] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:00.115356 6956 leader_election.cc:302] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:00.115546 7974 raft_consensus.cc:2781] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:00.115605 7974 raft_consensus.cc:482] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:00.115641 7974 raft_consensus.cc:3037] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.116547 7974 raft_consensus.cc:504] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:00.116905 7974 leader_election.cc:288] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:00.117312 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abb55c09be324f37a1a546e73008a80d" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:00.117349 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abb55c09be324f37a1a546e73008a80d" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:00.117442 7151 raft_consensus.cc:3037] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.117457 7279 raft_consensus.cc:3037] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.118225 7279 raft_consensus.cc:2454] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:00.118225 7151 raft_consensus.cc:2454] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:00.118573 6956 leader_election.cc:302] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:00.118757 7974 raft_consensus.cc:2781] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:00.118841 7974 raft_consensus.cc:686] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:00.118971 7974 consensus_queue.cc:237] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:00.120965 6738 catalog_manager.cc:5484] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:00.267787 8029 raft_consensus.cc:482] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.267939 8029 raft_consensus.cc:504] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.268421 8029 leader_election.cc:288] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:00.268977 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b794508aa77492398e48d25edfb8aa3" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:00.268996 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b794508aa77492398e48d25edfb8aa3" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:00.269124 7151 raft_consensus.cc:2454] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:00.269134 7279 raft_consensus.cc:2454] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:00.269454 6825 leader_election.cc:302] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:00.269661 8029 raft_consensus.cc:2781] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:00.269722 8029 raft_consensus.cc:482] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:00.269765 8029 raft_consensus.cc:3037] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.270880 8029 raft_consensus.cc:504] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.271242 8029 leader_election.cc:288] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:00.271631 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b794508aa77492398e48d25edfb8aa3" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:00.271754 7279 raft_consensus.cc:3037] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.271730 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b794508aa77492398e48d25edfb8aa3" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:00.271857 7151 raft_consensus.cc:3037] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.272609 7151 raft_consensus.cc:2454] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:00.272608 7279 raft_consensus.cc:2454] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:00.272920 6825 leader_election.cc:302] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:00.273104 8029 raft_consensus.cc:2781] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:00.273304 8029 raft_consensus.cc:686] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:00.273461 8029 consensus_queue.cc:237] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.275247 6738 catalog_manager.cc:5484] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
W20240503 07:59:00.448624 8134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:00.448959 8134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:00.452400 8134 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:59:00.459959 7969 consensus_queue.cc:1035] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.465092 7969 consensus_queue.cc:1035] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240503 07:59:00.537573 7151 raft_consensus.cc:1261] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.538242 7279 raft_consensus.cc:1261] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.538750 7974 consensus_queue.cc:1035] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.541347 8029 consensus_queue.cc:1035] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.542482 7151 raft_consensus.cc:1261] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.542516 7278 raft_consensus.cc:1261] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.543363 7974 consensus_queue.cc:1035] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.548556 8132 consensus_queue.cc:1035] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.554 ms
time per row: 0.0537769 ms
I20240503 07:59:00.610909 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:00.617269 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9a8ae5e91c469e81b58a61306d922c (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.617357 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9a8ae5e91c469e81b58a61306d922c (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.617632 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9a8ae5e91c469e81b58a61306d922c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.617673 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9a8ae5e91c469e81b58a61306d922c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.618624 7261 tablet_service.cc:1467] Processing CreateTablet for tablet f343ec9d35a940d991f6bc279a089bae (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.618624 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8483954eeb418db0e38021f6cb41d6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.618961 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f343ec9d35a940d991f6bc279a089bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.620196 8201 tablet_bootstrap.cc:492] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:00.620460 8200 tablet_bootstrap.cc:492] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:00.620884 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9a8ae5e91c469e81b58a61306d922c (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.621053 6868 tablet_service.cc:1467] Processing CreateTablet for tablet f343ec9d35a940d991f6bc279a089bae (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.621147 8201 tablet_bootstrap.cc:654] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.621167 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9a8ae5e91c469e81b58a61306d922c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.620884 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8483954eeb418db0e38021f6cb41d6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.621435 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8483954eeb418db0e38021f6cb41d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.621848 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8483954eeb418db0e38021f6cb41d6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.622022 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8483954eeb418db0e38021f6cb41d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.622428 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f343ec9d35a940d991f6bc279a089bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.623703 8204 tablet_bootstrap.cc:492] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:00.623986 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8483954eeb418db0e38021f6cb41d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.624737 8204 tablet_bootstrap.cc:654] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.624801 8200 tablet_bootstrap.cc:654] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.625895 7130 tablet_service.cc:1467] Processing CreateTablet for tablet f343ec9d35a940d991f6bc279a089bae (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.626175 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f343ec9d35a940d991f6bc279a089bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.626250 8202 tablet_bootstrap.cc:492] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:00.626849 8204 tablet_bootstrap.cc:492] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:00.626865 8201 tablet_bootstrap.cc:492] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:00.626935 8204 ts_tablet_manager.cc:1367] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:00.626952 8201 ts_tablet_manager.cc:1367] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:00.627233 8202 tablet_bootstrap.cc:654] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.627393 8204 raft_consensus.cc:348] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:00.627413 8201 raft_consensus.cc:348] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:00.627496 8204 raft_consensus.cc:374] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.627527 8201 raft_consensus.cc:374] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.627528 8204 raft_consensus.cc:729] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.627557 8201 raft_consensus.cc:729] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.627652 8204 consensus_queue.cc:260] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:00.627692 8201 consensus_queue.cc:260] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:00.627935 8204 ts_tablet_manager.cc:1398] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.627971 8201 ts_tablet_manager.cc:1398] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:00.628104 8204 tablet_bootstrap.cc:492] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:00.628142 8201 tablet_bootstrap.cc:492] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:00.628932 8204 tablet_bootstrap.cc:654] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.629048 8201 tablet_bootstrap.cc:654] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.629269 8200 tablet_bootstrap.cc:492] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:00.629352 8200 ts_tablet_manager.cc:1367] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:00.629835 8200 raft_consensus.cc:348] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:00.629945 8200 raft_consensus.cc:374] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.629976 8200 raft_consensus.cc:729] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.630115 8200 consensus_queue.cc:260] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:00.631820 8204 tablet_bootstrap.cc:492] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:00.631903 8204 ts_tablet_manager.cc:1367] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:00.631932 8200 ts_tablet_manager.cc:1398] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:00.631989 8201 tablet_bootstrap.cc:492] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:00.632063 8201 ts_tablet_manager.cc:1367] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:00.632357 8204 raft_consensus.cc:348] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.632475 8204 raft_consensus.cc:374] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.632510 8204 raft_consensus.cc:729] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.632513 8201 raft_consensus.cc:348] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.632630 8201 raft_consensus.cc:374] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.632625 8204 consensus_queue.cc:260] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.632679 8201 raft_consensus.cc:729] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.632835 8200 tablet_bootstrap.cc:492] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:00.632925 8204 ts_tablet_manager.cc:1398] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.632824 8201 consensus_queue.cc:260] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.633093 8204 tablet_bootstrap.cc:492] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:00.633276 8201 ts_tablet_manager.cc:1398] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.633749 8202 tablet_bootstrap.cc:492] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:00.633913 8202 ts_tablet_manager.cc:1367] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:00.634289 8204 tablet_bootstrap.cc:654] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.634835 8202 raft_consensus.cc:348] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.634991 8202 raft_consensus.cc:374] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.635042 8202 raft_consensus.cc:729] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.635183 8202 consensus_queue.cc:260] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.635468 8202 ts_tablet_manager.cc:1398] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.635641 8202 tablet_bootstrap.cc:492] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:00.636571 8202 tablet_bootstrap.cc:654] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.638013 8204 tablet_bootstrap.cc:492] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:00.638104 8204 ts_tablet_manager.cc:1367] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:00.638588 8204 raft_consensus.cc:348] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.638723 8204 raft_consensus.cc:374] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.638760 8204 raft_consensus.cc:729] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.638882 8204 consensus_queue.cc:260] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.639123 8200 tablet_bootstrap.cc:654] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.639187 8204 ts_tablet_manager.cc:1398] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.640372 8202 tablet_bootstrap.cc:492] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:00.640477 8202 ts_tablet_manager.cc:1367] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:00.640946 8202 raft_consensus.cc:348] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.641079 8202 raft_consensus.cc:374] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.641108 8202 raft_consensus.cc:729] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.641243 8202 consensus_queue.cc:260] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.641604 8202 ts_tablet_manager.cc:1398] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.641808 8200 tablet_bootstrap.cc:492] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:00.641911 8200 ts_tablet_manager.cc:1367] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 07:59:00.642367 8200 raft_consensus.cc:348] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.642470 8200 raft_consensus.cc:374] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.642490 8200 raft_consensus.cc:729] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.642602 8200 consensus_queue.cc:260] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.642858 8200 ts_tablet_manager.cc:1398] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.682466 8192 raft_consensus.cc:482] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.682631 8192 raft_consensus.cc:504] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.683101 8192 leader_election.cc:288] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:00.683565 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8483954eeb418db0e38021f6cb41d6" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:00.683576 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8483954eeb418db0e38021f6cb41d6" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:00.683707 7019 raft_consensus.cc:2454] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:00.683734 7279 raft_consensus.cc:2454] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:00.684082 6824 leader_election.cc:302] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:00.684274 8192 raft_consensus.cc:2781] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:00.684345 8192 raft_consensus.cc:482] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:00.684386 8192 raft_consensus.cc:3037] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.685261 8192 raft_consensus.cc:504] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.685624 8192 leader_election.cc:288] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:00.686015 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8483954eeb418db0e38021f6cb41d6" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:00.686132 7019 raft_consensus.cc:3037] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.686301 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8483954eeb418db0e38021f6cb41d6" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:00.686448 7279 raft_consensus.cc:3037] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.687215 7019 raft_consensus.cc:2454] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:00.687371 7279 raft_consensus.cc:2454] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:00.687539 6824 leader_election.cc:302] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:00.687702 8192 raft_consensus.cc:2781] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:00.687798 8192 raft_consensus.cc:686] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:00.687947 8192 consensus_queue.cc:237] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:00.689847 6746 catalog_manager.cc:5484] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:00.792016 8175 raft_consensus.cc:482] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.792171 8175 raft_consensus.cc:504] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:00.792662 8175 leader_election.cc:288] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:00.793239 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a8ae5e91c469e81b58a61306d922c" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:00.793239 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a8ae5e91c469e81b58a61306d922c" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:00.793414 6888 raft_consensus.cc:2454] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:00.793414 7151 raft_consensus.cc:2454] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:00.793761 6953 leader_election.cc:302] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:00.793957 8175 raft_consensus.cc:2781] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:00.794023 8175 raft_consensus.cc:482] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:00.794056 8175 raft_consensus.cc:3037] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.795042 8175 raft_consensus.cc:504] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:00.795408 8175 leader_election.cc:288] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:00.795830 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a8ae5e91c469e81b58a61306d922c" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:00.795886 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a8ae5e91c469e81b58a61306d922c" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:00.795955 7151 raft_consensus.cc:3037] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.796016 6888 raft_consensus.cc:3037] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.796926 7151 raft_consensus.cc:2454] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:00.796934 6888 raft_consensus.cc:2454] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:00.797231 6956 leader_election.cc:302] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:00.797422 8175 raft_consensus.cc:2781] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:00.797511 8175 raft_consensus.cc:686] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:00.797657 8175 consensus_queue.cc:237] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:00.799458 6746 catalog_manager.cc:5484] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:00.863330 8176 raft_consensus.cc:482] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.863485 8176 raft_consensus.cc:504] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.863958 8176 leader_election.cc:288] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:00.864405 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f343ec9d35a940d991f6bc279a089bae" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:00.864401 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f343ec9d35a940d991f6bc279a089bae" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:00.864542 7279 raft_consensus.cc:2454] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:00.864548 6888 raft_consensus.cc:2454] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:00.864864 7084 leader_election.cc:302] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:00.865121 8176 raft_consensus.cc:2781] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:00.865192 8176 raft_consensus.cc:482] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:00.865229 8176 raft_consensus.cc:3037] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.866135 8176 raft_consensus.cc:504] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.866600 8176 leader_election.cc:288] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:00.867080 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f343ec9d35a940d991f6bc279a089bae" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:00.867079 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f343ec9d35a940d991f6bc279a089bae" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:00.867206 7279 raft_consensus.cc:3037] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.867210 6888 raft_consensus.cc:3037] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.868005 7279 raft_consensus.cc:2454] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:00.868018 6888 raft_consensus.cc:2454] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:00.868335 7087 leader_election.cc:302] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:00.868505 8176 raft_consensus.cc:2781] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:00.868600 8176 raft_consensus.cc:686] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:00.868778 8176 consensus_queue.cc:237] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:00.870579 6746 catalog_manager.cc:5484] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
W20240503 07:59:01.041281 8216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:01.041568 8216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:01.044984 8216 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 07:59:01.140851 8192 consensus_queue.cc:1035] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.147835 7278 raft_consensus.cc:1261] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.147941 6888 raft_consensus.cc:1261] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.148305 8176 consensus_queue.cc:1035] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.148494 7969 consensus_queue.cc:1035] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.152464 6889 raft_consensus.cc:1261] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.153282 7151 raft_consensus.cc:1261] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.153703 8132 consensus_queue.cc:1035] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.153817 7974 consensus_queue.cc:1035] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.154402 7974 consensus_queue.cc:1035] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 133.416 ms
time per row: 0.0667079 ms
I20240503 07:59:01.224452 6746 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:01.230620 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 31c76213f3ec451084ae7b0551cd7b48 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.230823 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 31c76213f3ec451084ae7b0551cd7b48 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.230985 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c76213f3ec451084ae7b0551cd7b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.230937 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 31c76213f3ec451084ae7b0551cd7b48 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.231135 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c76213f3ec451084ae7b0551cd7b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.231199 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c76213f3ec451084ae7b0551cd7b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.233546 7261 tablet_service.cc:1467] Processing CreateTablet for tablet dd6aa70d9c4c4baea597bbb9788fdb37 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.233625 8284 tablet_bootstrap.cc:492] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:01.233830 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6aa70d9c4c4baea597bbb9788fdb37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.234012 7130 tablet_service.cc:1467] Processing CreateTablet for tablet dd6aa70d9c4c4baea597bbb9788fdb37 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.234275 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6aa70d9c4c4baea597bbb9788fdb37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.234663 8283 tablet_bootstrap.cc:492] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:01.235066 8284 tablet_bootstrap.cc:654] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.235599 8283 tablet_bootstrap.cc:654] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.236011 8282 tablet_bootstrap.cc:492] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:01.236732 8283 tablet_bootstrap.cc:492] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:01.236829 8283 ts_tablet_manager.cc:1367] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:01.236956 6999 tablet_service.cc:1467] Processing CreateTablet for tablet dd6aa70d9c4c4baea597bbb9788fdb37 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.237190 8282 tablet_bootstrap.cc:654] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.237217 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6aa70d9c4c4baea597bbb9788fdb37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.237298 8283 raft_consensus.cc:348] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.237412 8283 raft_consensus.cc:374] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.237388 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 3658573037664f03b38fefd31d2fe587 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.237447 8283 raft_consensus.cc:729] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.237583 8283 consensus_queue.cc:260] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.237644 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3658573037664f03b38fefd31d2fe587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.237891 8283 ts_tablet_manager.cc:1398] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.238073 8283 tablet_bootstrap.cc:492] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:01.238950 8283 tablet_bootstrap.cc:654] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.239735 8282 tablet_bootstrap.cc:492] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:01.239836 8282 ts_tablet_manager.cc:1367] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:01.240183 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 3658573037664f03b38fefd31d2fe587 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.240439 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3658573037664f03b38fefd31d2fe587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.240541 8282 raft_consensus.cc:348] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.240655 8282 raft_consensus.cc:374] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.240689 8282 raft_consensus.cc:729] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.240814 8282 consensus_queue.cc:260] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.241068 8282 ts_tablet_manager.cc:1398] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:01.241200 8282 tablet_bootstrap.cc:492] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:01.241890 8282 tablet_bootstrap.cc:654] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.242266 8289 tablet_bootstrap.cc:492] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:01.239748 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 3658573037664f03b38fefd31d2fe587 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.243141 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3658573037664f03b38fefd31d2fe587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.243803 8289 tablet_bootstrap.cc:654] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.245095 8289 tablet_bootstrap.cc:492] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:01.245106 8284 tablet_bootstrap.cc:492] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:01.245204 8284 ts_tablet_manager.cc:1367] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:01.245205 8289 ts_tablet_manager.cc:1367] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:01.245456 8282 tablet_bootstrap.cc:492] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:01.245537 8282 ts_tablet_manager.cc:1367] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:01.245684 8289 raft_consensus.cc:348] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:01.245703 8284 raft_consensus.cc:348] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.245805 8284 raft_consensus.cc:374] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.245816 8289 raft_consensus.cc:374] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.245834 8284 raft_consensus.cc:729] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.245851 8289 raft_consensus.cc:729] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.245944 8284 consensus_queue.cc:260] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.245960 8282 raft_consensus.cc:348] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.246042 8282 raft_consensus.cc:374] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.246071 8282 raft_consensus.cc:729] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.246016 8289 consensus_queue.cc:260] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:01.246093 8283 tablet_bootstrap.cc:492] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:01.246176 8283 ts_tablet_manager.cc:1367] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:01.246189 8282 consensus_queue.cc:260] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.246253 8284 ts_tablet_manager.cc:1398] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.246306 8289 ts_tablet_manager.cc:1398] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.246448 8284 tablet_bootstrap.cc:492] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:01.246471 8282 ts_tablet_manager.cc:1398] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:01.246626 8282 tablet_bootstrap.cc:492] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:01.246642 8283 raft_consensus.cc:348] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.246750 8283 raft_consensus.cc:374] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.246778 8283 raft_consensus.cc:729] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.246893 8283 consensus_queue.cc:260] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.247170 8283 ts_tablet_manager.cc:1398] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.247417 8284 tablet_bootstrap.cc:654] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.247417 8282 tablet_bootstrap.cc:654] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.249683 8282 tablet_bootstrap.cc:492] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:01.249830 8282 ts_tablet_manager.cc:1367] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:01.249928 8284 tablet_bootstrap.cc:492] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:01.250015 8284 ts_tablet_manager.cc:1367] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:01.250465 8282 raft_consensus.cc:348] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:01.250465 8284 raft_consensus.cc:348] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.250576 8284 raft_consensus.cc:374] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.250612 8284 raft_consensus.cc:729] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.250697 8282 raft_consensus.cc:374] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.250770 8282 raft_consensus.cc:729] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.250764 8284 consensus_queue.cc:260] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.250980 8282 consensus_queue.cc:260] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:01.251331 8282 ts_tablet_manager.cc:1398] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.251426 8284 ts_tablet_manager.cc:1398] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.252126 8284 tablet_bootstrap.cc:492] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:01.253203 8284 tablet_bootstrap.cc:654] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.254739 8284 tablet_bootstrap.cc:492] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:01.254869 8284 ts_tablet_manager.cc:1367] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:01.255324 8284 raft_consensus.cc:348] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:01.255437 8284 raft_consensus.cc:374] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.255468 8284 raft_consensus.cc:729] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.255571 8284 consensus_queue.cc:260] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:01.255839 8284 ts_tablet_manager.cc:1398] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:01.293359 7974 raft_consensus.cc:482] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:01.293495 7974 raft_consensus.cc:504] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.293886 7974 leader_election.cc:288] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:01.294481 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c76213f3ec451084ae7b0551cd7b48" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:01.294481 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c76213f3ec451084ae7b0551cd7b48" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:01.294651 7151 raft_consensus.cc:2454] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:01.294651 7278 raft_consensus.cc:2454] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:01.294957 6956 leader_election.cc:302] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:01.295202 7974 raft_consensus.cc:2781] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:01.295265 7974 raft_consensus.cc:482] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:01.295306 7974 raft_consensus.cc:3037] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.296195 7974 raft_consensus.cc:504] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.296532 7974 leader_election.cc:288] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:01.296963 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c76213f3ec451084ae7b0551cd7b48" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:01.297082 7151 raft_consensus.cc:3037] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.297874 7151 raft_consensus.cc:2454] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:01.298135 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c76213f3ec451084ae7b0551cd7b48" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:01.298203 6956 leader_election.cc:302] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:01.298265 7278 raft_consensus.cc:3037] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.298460 7974 raft_consensus.cc:2781] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:01.298561 7974 raft_consensus.cc:686] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:01.298700 7974 consensus_queue.cc:237] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.299257 7278 raft_consensus.cc:2454] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:01.300761 6745 catalog_manager.cc:5484] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:01.320020 8257 raft_consensus.cc:482] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:01.320176 8257 raft_consensus.cc:504] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.321316 8257 leader_election.cc:288] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:01.325364 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6aa70d9c4c4baea597bbb9788fdb37" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:01.325392 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6aa70d9c4c4baea597bbb9788fdb37" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:01.325531 7151 raft_consensus.cc:2454] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:01.325538 7019 raft_consensus.cc:2454] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:01.325933 7218 leader_election.cc:302] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:01.326246 8257 raft_consensus.cc:2781] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:01.326308 8257 raft_consensus.cc:482] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:01.326383 8257 raft_consensus.cc:3037] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.327296 8257 raft_consensus.cc:504] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.327683 8257 leader_election.cc:288] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:01.328085 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6aa70d9c4c4baea597bbb9788fdb37" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:01.328070 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6aa70d9c4c4baea597bbb9788fdb37" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:01.328207 7151 raft_consensus.cc:3037] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.328212 7019 raft_consensus.cc:3037] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.328979 7151 raft_consensus.cc:2454] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:01.329301 7218 leader_election.cc:302] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:01.329486 8257 raft_consensus.cc:2781] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:01.329605 8257 raft_consensus.cc:686] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:01.329926 8257 consensus_queue.cc:237] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.332326 6745 catalog_manager.cc:5484] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:01.351367 7019 raft_consensus.cc:2454] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:01.371026 8132 raft_consensus.cc:482] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:01.371181 8132 raft_consensus.cc:504] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:01.371646 8132 leader_election.cc:288] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:01.372121 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3658573037664f03b38fefd31d2fe587" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:01.372148 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3658573037664f03b38fefd31d2fe587" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:01.372267 7151 raft_consensus.cc:2454] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:01.372313 7019 raft_consensus.cc:2454] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:01.372637 6825 leader_election.cc:302] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:01.372839 8132 raft_consensus.cc:2781] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:01.372900 8132 raft_consensus.cc:482] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:01.372936 8132 raft_consensus.cc:3037] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.373766 8132 raft_consensus.cc:504] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:01.374172 8132 leader_election.cc:288] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:01.374578 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3658573037664f03b38fefd31d2fe587" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:01.374588 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3658573037664f03b38fefd31d2fe587" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:01.374707 7151 raft_consensus.cc:3037] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.374712 7019 raft_consensus.cc:3037] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.375504 7019 raft_consensus.cc:2454] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:01.375692 7151 raft_consensus.cc:2454] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:01.375909 6824 leader_election.cc:302] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:01.376116 8132 raft_consensus.cc:2781] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:01.376214 8132 raft_consensus.cc:686] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:01.376364 8132 consensus_queue.cc:237] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:01.378159 6745 catalog_manager.cc:5484] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
W20240503 07:59:01.513669 8300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:01.513973 8300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:01.517293 8300 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 07:59:01.605515 7151 raft_consensus.cc:1261] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.605810 7019 raft_consensus.cc:1261] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.606227 8132 consensus_queue.cc:1035] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.606639 7278 raft_consensus.cc:1261] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:01.607065 8257 consensus_queue.cc:1035] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.607223 7151 raft_consensus.cc:1261] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:01.607838 8328 consensus_queue.cc:1035] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.607928 7150 raft_consensus.cc:1261] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.608834 8175 consensus_queue.cc:1035] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.614518 8253 consensus_queue.cc:1035] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.616300 7020 raft_consensus.cc:1261] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.616775 8132 consensus_queue.cc:1035] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.709 ms
time per row: 0.0538547 ms
I20240503 07:59:01.674995 6747 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:01.683530 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 77a24f5c59ec4821b56163a6dd685dc9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.683967 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a24f5c59ec4821b56163a6dd685dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.684499 6999 tablet_service.cc:1467] Processing CreateTablet for tablet a8db24c1bdb142c2b2a6b3041f308892 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.684718 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 77a24f5c59ec4821b56163a6dd685dc9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.684818 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8db24c1bdb142c2b2a6b3041f308892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.684988 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a24f5c59ec4821b56163a6dd685dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.686509 8283 tablet_bootstrap.cc:492] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:01.686753 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 77a24f5c59ec4821b56163a6dd685dc9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.687029 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a24f5c59ec4821b56163a6dd685dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.687469 8283 tablet_bootstrap.cc:654] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.687429 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2c2c5aafd640ccac547de40ec34841 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.687637 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2c2c5aafd640ccac547de40ec34841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.688705 8283 tablet_bootstrap.cc:492] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:01.688809 8283 ts_tablet_manager.cc:1367] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:01.689270 8283 raft_consensus.cc:348] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:01.689397 8283 raft_consensus.cc:374] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.689426 8283 raft_consensus.cc:729] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.689417 6869 tablet_service.cc:1467] Processing CreateTablet for tablet a8db24c1bdb142c2b2a6b3041f308892 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.689534 8283 consensus_queue.cc:260] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:01.689663 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8db24c1bdb142c2b2a6b3041f308892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.689826 8282 tablet_bootstrap.cc:492] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:01.690162 8283 ts_tablet_manager.cc:1398] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.690732 8282 tablet_bootstrap.cc:654] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.691442 8289 tablet_bootstrap.cc:492] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:01.692271 8289 tablet_bootstrap.cc:654] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.692245 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2c2c5aafd640ccac547de40ec34841 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.692520 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2c2c5aafd640ccac547de40ec34841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.693816 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2c2c5aafd640ccac547de40ec34841 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.694089 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2c2c5aafd640ccac547de40ec34841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.694662 7260 tablet_service.cc:1467] Processing CreateTablet for tablet a8db24c1bdb142c2b2a6b3041f308892 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.694912 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8db24c1bdb142c2b2a6b3041f308892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.694933 8284 tablet_bootstrap.cc:492] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:01.695811 8284 tablet_bootstrap.cc:654] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.696520 8283 tablet_bootstrap.cc:492] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:01.697783 8283 tablet_bootstrap.cc:654] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.697948 8284 tablet_bootstrap.cc:492] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:01.697970 8282 tablet_bootstrap.cc:492] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:01.698045 8284 ts_tablet_manager.cc:1367] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:01.698064 8282 ts_tablet_manager.cc:1367] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:01.698576 8284 raft_consensus.cc:348] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.698712 8284 raft_consensus.cc:374] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.698745 8284 raft_consensus.cc:729] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.698727 8282 raft_consensus.cc:348] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.698838 8282 raft_consensus.cc:374] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.698875 8282 raft_consensus.cc:729] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.698879 8284 consensus_queue.cc:260] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.699002 8282 consensus_queue.cc:260] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.699164 8284 ts_tablet_manager.cc:1398] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.699335 8282 ts_tablet_manager.cc:1398] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.699510 8282 tablet_bootstrap.cc:492] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:01.700181 8283 tablet_bootstrap.cc:492] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:01.700278 8283 ts_tablet_manager.cc:1367] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:01.700477 8282 tablet_bootstrap.cc:654] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.702631 8283 raft_consensus.cc:348] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.702775 8283 raft_consensus.cc:374] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.702817 8283 raft_consensus.cc:729] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.702956 8283 consensus_queue.cc:260] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.703357 8283 ts_tablet_manager.cc:1398] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:01.703547 8283 tablet_bootstrap.cc:492] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:01.705312 8283 tablet_bootstrap.cc:654] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.705824 8282 tablet_bootstrap.cc:492] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:01.705932 8282 ts_tablet_manager.cc:1367] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:01.706404 8282 raft_consensus.cc:348] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:01.706526 8282 raft_consensus.cc:374] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.706560 8282 raft_consensus.cc:729] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.706689 8282 consensus_queue.cc:260] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:01.706995 8282 ts_tablet_manager.cc:1398] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.707537 8283 tablet_bootstrap.cc:492] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:01.707540 8289 tablet_bootstrap.cc:492] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:01.707646 8289 ts_tablet_manager.cc:1367] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 07:59:01.707664 8283 ts_tablet_manager.cc:1367] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:01.708112 8283 raft_consensus.cc:348] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.708113 8289 raft_consensus.cc:348] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:01.708218 8283 raft_consensus.cc:374] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.708218 8289 raft_consensus.cc:374] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.708249 8283 raft_consensus.cc:729] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.708249 8289 raft_consensus.cc:729] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.708361 8283 consensus_queue.cc:260] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.708384 8289 consensus_queue.cc:260] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:01.708658 8289 ts_tablet_manager.cc:1398] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.708699 8283 ts_tablet_manager.cc:1398] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.708843 8289 tablet_bootstrap.cc:492] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:01.709793 8289 tablet_bootstrap.cc:654] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.716117 8289 tablet_bootstrap.cc:492] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:01.716259 8289 ts_tablet_manager.cc:1367] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 07:59:01.716715 8289 raft_consensus.cc:348] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.716830 8289 raft_consensus.cc:374] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.716850 8289 raft_consensus.cc:729] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.716976 8289 consensus_queue.cc:260] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.717257 8289 ts_tablet_manager.cc:1398] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:01.717442 8289 tablet_bootstrap.cc:492] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:01.718292 8289 tablet_bootstrap.cc:654] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.719480 8289 tablet_bootstrap.cc:492] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:01.719565 8289 ts_tablet_manager.cc:1367] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:01.719987 8289 raft_consensus.cc:348] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.720139 8289 raft_consensus.cc:374] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.720209 8289 raft_consensus.cc:729] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.720352 8289 consensus_queue.cc:260] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.720654 8289 ts_tablet_manager.cc:1398] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.737296 8257 raft_consensus.cc:482] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:01.737437 8257 raft_consensus.cc:504] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:01.737951 8257 leader_election.cc:288] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:01.738408 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a24f5c59ec4821b56163a6dd685dc9" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:01.738574 7019 raft_consensus.cc:2454] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:01.738924 7217 leader_election.cc:302] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:01.739141 8257 raft_consensus.cc:2781] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:01.739223 8257 raft_consensus.cc:482] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:01.739260 8257 raft_consensus.cc:3037] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.740227 8257 raft_consensus.cc:504] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:01.740636 8257 leader_election.cc:288] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:01.740991 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a24f5c59ec4821b56163a6dd685dc9" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:01.741159 7019 raft_consensus.cc:3037] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.742252 7019 raft_consensus.cc:2454] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:01.742775 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a24f5c59ec4821b56163a6dd685dc9" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:01.742818 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a24f5c59ec4821b56163a6dd685dc9" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:01.742902 6889 raft_consensus.cc:2454] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:01.742949 7217 leader_election.cc:302] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:01.743446 8257 raft_consensus.cc:2781] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:01.743541 8257 raft_consensus.cc:686] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:01.743693 8257 consensus_queue.cc:237] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:01.745561 6745 catalog_manager.cc:5484] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:01.786055 8176 raft_consensus.cc:482] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:01.786208 8176 raft_consensus.cc:504] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.786695 8176 leader_election.cc:288] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:01.787161 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2c2c5aafd640ccac547de40ec34841" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:01.787212 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2c2c5aafd640ccac547de40ec34841" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:01.787344 7279 raft_consensus.cc:2454] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:01.787338 6888 raft_consensus.cc:2454] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:01.787685 7084 leader_election.cc:302] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:01.787863 8176 raft_consensus.cc:2781] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:01.787926 8176 raft_consensus.cc:482] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:01.787966 8176 raft_consensus.cc:3037] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.788846 8176 raft_consensus.cc:504] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.789220 8176 leader_election.cc:288] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:01.789598 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2c2c5aafd640ccac547de40ec34841" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:01.789731 6888 raft_consensus.cc:3037] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.789769 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2c2c5aafd640ccac547de40ec34841" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:01.789901 7279 raft_consensus.cc:3037] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.790623 6888 raft_consensus.cc:2454] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:01.790786 7279 raft_consensus.cc:2454] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:01.790961 7084 leader_election.cc:302] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:01.791163 8176 raft_consensus.cc:2781] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:01.791353 8176 raft_consensus.cc:686] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:01.791512 8176 consensus_queue.cc:237] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:01.793429 6745 catalog_manager.cc:5484] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:01.992810 8329 raft_consensus.cc:482] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:01.992964 8329 raft_consensus.cc:504] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.993448 8329 leader_election.cc:288] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:01.993976 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8db24c1bdb142c2b2a6b3041f308892" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:01.993995 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8db24c1bdb142c2b2a6b3041f308892" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:01.994132 7279 raft_consensus.cc:2454] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:01.994211 6888 raft_consensus.cc:2454] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:01.994534 6956 leader_election.cc:302] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:01.994743 8329 raft_consensus.cc:2781] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:01.994822 8329 raft_consensus.cc:482] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:01.994869 8329 raft_consensus.cc:3037] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.995819 8329 raft_consensus.cc:504] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:01.996171 8329 leader_election.cc:288] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:01.996568 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8db24c1bdb142c2b2a6b3041f308892" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:01.996574 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8db24c1bdb142c2b2a6b3041f308892" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:01.996686 6888 raft_consensus.cc:3037] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.996692 7279 raft_consensus.cc:3037] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.997498 7279 raft_consensus.cc:2454] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:01.997515 6888 raft_consensus.cc:2454] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:01.997813 6953 leader_election.cc:302] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:01.998018 8329 raft_consensus.cc:2781] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:01.998116 8329 raft_consensus.cc:686] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:01.998482 8329 consensus_queue.cc:237] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:02.000299 6745 catalog_manager.cc:5484] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:02.183805 8176 consensus_queue.cc:1035] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.188360 8176 consensus_queue.cc:1035] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:02.194067 8349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:02.194432 8349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:02.197885 8349 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 07:59:02.245790 8257 consensus_queue.cc:1035] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.251220 6888 raft_consensus.cc:3037] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.253041 8257 consensus_queue.cc:1035] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.324760 6888 raft_consensus.cc:1261] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.325345 7279 raft_consensus.cc:1261] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.325368 8329 consensus_queue.cc:1035] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.326308 8329 consensus_queue.cc:1035] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.342689 8352 mvcc.cc:205] Tried to move back new op lower bound from 7023505990956150784 to 7023505989627420672. Current Snapshot: MvccSnapshot[applied={T|T < 7023505990956150784}]
I20240503 07:59:02.348016 8380 mvcc.cc:205] Tried to move back new op lower bound from 7023505991022419968 to 7023505989627420672. Current Snapshot: MvccSnapshot[applied={T|T < 7023505990956150784}]
INSERT report
rows total: 2000
time total: 121.71 ms
time per row: 0.0608548 ms
I20240503 07:59:02.366106 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:02.372789 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 7b424a8b7bba44b098731f8ee18e8c9b (DEFAULT_TABLE table=rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.373041 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 7b424a8b7bba44b098731f8ee18e8c9b (DEFAULT_TABLE table=rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.373153 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b424a8b7bba44b098731f8ee18e8c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.373375 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b424a8b7bba44b098731f8ee18e8c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.373842 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 5390e70b0449449aa3af05112bd29797 (DEFAULT_TABLE table=rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.374137 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5390e70b0449449aa3af05112bd29797. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.374893 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 5390e70b0449449aa3af05112bd29797 (DEFAULT_TABLE table=rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.375140 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5390e70b0449449aa3af05112bd29797. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.375960 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 7e8e04c50d7147af903d7113df3d9744 (DEFAULT_TABLE table=rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.376207 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8e04c50d7147af903d7113df3d9744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.376149 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 5390e70b0449449aa3af05112bd29797 (DEFAULT_TABLE table=rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.376425 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5390e70b0449449aa3af05112bd29797. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.373842 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 7b424a8b7bba44b098731f8ee18e8c9b (DEFAULT_TABLE table=rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.377436 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b424a8b7bba44b098731f8ee18e8c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.378798 8410 tablet_bootstrap.cc:492] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:02.378990 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 7e8e04c50d7147af903d7113df3d9744 (DEFAULT_TABLE table=rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.379240 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8e04c50d7147af903d7113df3d9744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.380019 8410 tablet_bootstrap.cc:654] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.380156 8409 tablet_bootstrap.cc:492] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:02.381768 8409 tablet_bootstrap.cc:654] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.382524 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 7e8e04c50d7147af903d7113df3d9744 (DEFAULT_TABLE table=rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.382745 8410 tablet_bootstrap.cc:492] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:02.382824 8410 ts_tablet_manager.cc:1367] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:02.382828 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8e04c50d7147af903d7113df3d9744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.383116 8409 tablet_bootstrap.cc:492] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:02.383206 8409 ts_tablet_manager.cc:1367] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:02.383292 8410 raft_consensus.cc:348] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.383397 8410 raft_consensus.cc:374] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.383422 8410 raft_consensus.cc:729] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.383553 8410 consensus_queue.cc:260] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.383653 8409 raft_consensus.cc:348] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.383757 8409 raft_consensus.cc:374] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.383785 8409 raft_consensus.cc:729] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.383805 8410 ts_tablet_manager.cc:1398] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.383900 8409 consensus_queue.cc:260] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.383962 8410 tablet_bootstrap.cc:492] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:02.384157 8409 ts_tablet_manager.cc:1398] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:02.384348 8409 tablet_bootstrap.cc:492] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:02.384819 8410 tablet_bootstrap.cc:654] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.385202 8409 tablet_bootstrap.cc:654] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.386857 8410 tablet_bootstrap.cc:492] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:02.386942 8410 ts_tablet_manager.cc:1367] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:02.387180 8409 tablet_bootstrap.cc:492] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:02.387260 8409 ts_tablet_manager.cc:1367] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:02.387624 8410 raft_consensus.cc:348] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.387753 8410 raft_consensus.cc:374] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.387784 8410 raft_consensus.cc:729] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.387797 8407 tablet_bootstrap.cc:492] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:02.387753 8409 raft_consensus.cc:348] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.387848 8409 raft_consensus.cc:374] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.387873 8409 raft_consensus.cc:729] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.387903 8410 consensus_queue.cc:260] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.388005 8409 consensus_queue.cc:260] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.388191 8410 ts_tablet_manager.cc:1398] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.388375 8409 ts_tablet_manager.cc:1398] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:02.388713 8407 tablet_bootstrap.cc:654] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.389782 8408 tablet_bootstrap.cc:492] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:02.390269 8407 tablet_bootstrap.cc:492] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:02.390413 8407 ts_tablet_manager.cc:1367] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:02.390733 8408 tablet_bootstrap.cc:654] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.390889 8407 raft_consensus.cc:348] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.391000 8407 raft_consensus.cc:374] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.391031 8407 raft_consensus.cc:729] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.391198 8407 consensus_queue.cc:260] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.391492 8407 ts_tablet_manager.cc:1398] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:02.391731 8407 tablet_bootstrap.cc:492] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:02.392630 8407 tablet_bootstrap.cc:654] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.393945 8407 tablet_bootstrap.cc:492] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:02.394469 8407 ts_tablet_manager.cc:1367] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:02.394943 8407 raft_consensus.cc:348] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.395084 8407 raft_consensus.cc:374] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.395120 8407 raft_consensus.cc:729] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.395265 8407 consensus_queue.cc:260] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.395548 8407 ts_tablet_manager.cc:1398] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.395771 8407 tablet_bootstrap.cc:492] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:02.396765 8407 tablet_bootstrap.cc:654] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.397969 8407 tablet_bootstrap.cc:492] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:02.398053 8407 ts_tablet_manager.cc:1367] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:02.398504 8407 raft_consensus.cc:348] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.398612 8407 raft_consensus.cc:374] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.398638 8407 raft_consensus.cc:729] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.398751 8407 consensus_queue.cc:260] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.398876 8408 tablet_bootstrap.cc:492] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:02.398994 8408 ts_tablet_manager.cc:1367] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:02.399385 8407 ts_tablet_manager.cc:1398] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.399529 8408 raft_consensus.cc:348] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.399643 8408 raft_consensus.cc:374] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.399675 8408 raft_consensus.cc:729] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.399807 8408 consensus_queue.cc:260] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.400650 8408 ts_tablet_manager.cc:1398] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:02.400830 8408 tablet_bootstrap.cc:492] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:02.401878 8408 tablet_bootstrap.cc:654] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.404174 8408 tablet_bootstrap.cc:492] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:02.404273 8408 ts_tablet_manager.cc:1367] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:02.404729 8408 raft_consensus.cc:348] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.404850 8408 raft_consensus.cc:374] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.404877 8408 raft_consensus.cc:729] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.404989 8408 consensus_queue.cc:260] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.405704 8408 ts_tablet_manager.cc:1398] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.416846 8176 raft_consensus.cc:482] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:02.417001 8176 raft_consensus.cc:504] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.417508 8176 leader_election.cc:288] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:02.418051 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5390e70b0449449aa3af05112bd29797" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:02.418164 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5390e70b0449449aa3af05112bd29797" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:02.418251 6888 raft_consensus.cc:2454] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:02.418329 7279 raft_consensus.cc:2454] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:02.418663 7084 leader_election.cc:302] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:02.419008 8176 raft_consensus.cc:2781] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:02.419073 8176 raft_consensus.cc:482] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:02.419108 8176 raft_consensus.cc:3037] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.420069 8176 raft_consensus.cc:504] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.420437 8176 leader_election.cc:288] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:02.420758 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5390e70b0449449aa3af05112bd29797" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:02.420886 6888 raft_consensus.cc:3037] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.420912 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5390e70b0449449aa3af05112bd29797" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:02.421054 7279 raft_consensus.cc:3037] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.421731 6888 raft_consensus.cc:2454] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:02.422026 7084 leader_election.cc:302] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:02.422109 7279 raft_consensus.cc:2454] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:02.422210 8176 raft_consensus.cc:2781] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:02.422286 8176 raft_consensus.cc:686] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:02.422470 8176 consensus_queue.cc:237] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.424628 6740 catalog_manager.cc:5484] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:02.451129 7969 raft_consensus.cc:482] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:02.451269 7969 raft_consensus.cc:504] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.451757 7969 leader_election.cc:288] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:02.452284 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8e04c50d7147af903d7113df3d9744" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:02.452342 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8e04c50d7147af903d7113df3d9744" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:02.452438 7279 raft_consensus.cc:2454] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:02.452523 7019 raft_consensus.cc:2454] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:02.452769 7087 leader_election.cc:302] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:02.452986 7969 raft_consensus.cc:2781] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:02.453050 7969 raft_consensus.cc:482] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:02.453080 7969 raft_consensus.cc:3037] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.453967 7969 raft_consensus.cc:504] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.454438 7969 leader_election.cc:288] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:02.454870 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8e04c50d7147af903d7113df3d9744" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:02.455021 7279 raft_consensus.cc:3037] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.455050 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8e04c50d7147af903d7113df3d9744" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:02.455211 7019 raft_consensus.cc:3037] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.455991 7279 raft_consensus.cc:2454] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:02.456358 7019 raft_consensus.cc:2454] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:02.456718 7086 leader_election.cc:302] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:02.456992 7969 raft_consensus.cc:2781] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:02.457085 7969 raft_consensus.cc:686] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:02.457243 7969 consensus_queue.cc:237] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:02.459594 6740 catalog_manager.cc:5484] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:02.564992 8328 raft_consensus.cc:482] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:02.565171 8328 raft_consensus.cc:504] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.565742 8328 leader_election.cc:288] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:02.566505 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b424a8b7bba44b098731f8ee18e8c9b" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:02.566524 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b424a8b7bba44b098731f8ee18e8c9b" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:02.566684 6888 raft_consensus.cc:2454] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:02.566694 7149 raft_consensus.cc:2454] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:02.567111 6956 leader_election.cc:302] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:02.567322 8328 raft_consensus.cc:2781] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:02.567381 8328 raft_consensus.cc:482] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:02.567420 8328 raft_consensus.cc:3037] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.568496 8328 raft_consensus.cc:504] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.569270 8328 leader_election.cc:288] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:02.569408 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b424a8b7bba44b098731f8ee18e8c9b" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:02.569561 6888 raft_consensus.cc:3037] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.570731 6888 raft_consensus.cc:2454] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:02.571174 6953 leader_election.cc:302] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:02.571154 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b424a8b7bba44b098731f8ee18e8c9b" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:02.571305 7149 raft_consensus.cc:3037] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.571435 8328 raft_consensus.cc:2781] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:02.571790 8328 raft_consensus.cc:686] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:02.571991 8328 consensus_queue.cc:237] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.572419 7149 raft_consensus.cc:2454] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:02.574719 6740 catalog_manager.cc:5484] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
W20240503 07:59:02.740733 8423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:02.741043 8423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:02.744457 8423 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 07:59:02.809957 7969 consensus_queue.cc:1035] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.814998 8176 consensus_queue.cc:1035] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.832393 6888 raft_consensus.cc:1261] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.832868 8329 consensus_queue.cc:1035] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.835067 7149 raft_consensus.cc:1261] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.839171 8329 consensus_queue.cc:1035] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.839419 7020 raft_consensus.cc:1261] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.839591 7279 raft_consensus.cc:1261] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.841398 8176 consensus_queue.cc:1035] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.842795 7969 consensus_queue.cc:1035] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.853212 8448 mvcc.cc:205] Tried to move back new op lower bound from 7023505993064771584 to 7023505991978246144. Current Snapshot: MvccSnapshot[applied={T|T < 7023505993031208960}]
I20240503 07:59:02.866392 8392 mvcc.cc:205] Tried to move back new op lower bound from 7023505993031208960 to 7023505991978246144. Current Snapshot: MvccSnapshot[applied={T|T < 7023505993031208960}]
INSERT report
rows total: 2000
time total: 118.106 ms
time per row: 0.0590531 ms
I20240503 07:59:02.910126 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:02.916589 7260 tablet_service.cc:1467] Processing CreateTablet for tablet fccdd597c5b54f41bb846d933a1e9a84 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.916687 6869 tablet_service.cc:1467] Processing CreateTablet for tablet fccdd597c5b54f41bb846d933a1e9a84 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.916843 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 91af7b042df146e980638f0f79bde295 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.916972 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fccdd597c5b54f41bb846d933a1e9a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.916985 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fccdd597c5b54f41bb846d933a1e9a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.917439 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91af7b042df146e980638f0f79bde295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.917585 6867 tablet_service.cc:1467] Processing CreateTablet for tablet b916e89edaf046b99767e5cca531bb8b (DEFAULT_TABLE table=rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.917789 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b916e89edaf046b99767e5cca531bb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.919508 8470 tablet_bootstrap.cc:492] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:02.919924 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 91af7b042df146e980638f0f79bde295 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.920020 7131 tablet_service.cc:1467] Processing CreateTablet for tablet b916e89edaf046b99767e5cca531bb8b (DEFAULT_TABLE table=rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.920223 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91af7b042df146e980638f0f79bde295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.920399 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b916e89edaf046b99767e5cca531bb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.920425 6998 tablet_service.cc:1467] Processing CreateTablet for tablet fccdd597c5b54f41bb846d933a1e9a84 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.920670 8470 tablet_bootstrap.cc:654] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.920699 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fccdd597c5b54f41bb846d933a1e9a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.920660 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 91af7b042df146e980638f0f79bde295 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.922441 8473 tablet_bootstrap.cc:492] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:02.922556 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91af7b042df146e980638f0f79bde295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.923324 8473 tablet_bootstrap.cc:654] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.923357 7260 tablet_service.cc:1467] Processing CreateTablet for tablet b916e89edaf046b99767e5cca531bb8b (DEFAULT_TABLE table=rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.923614 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b916e89edaf046b99767e5cca531bb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.924134 8471 tablet_bootstrap.cc:492] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:02.924882 8474 tablet_bootstrap.cc:492] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:02.925087 8471 tablet_bootstrap.cc:654] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.925838 8474 tablet_bootstrap.cc:654] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.926705 8470 tablet_bootstrap.cc:492] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:02.926870 8470 ts_tablet_manager.cc:1367] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:02.927160 8474 tablet_bootstrap.cc:492] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:02.927268 8474 ts_tablet_manager.cc:1367] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:02.927611 8471 tablet_bootstrap.cc:492] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:02.927709 8471 ts_tablet_manager.cc:1367] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:02.927582 8470 raft_consensus.cc:348] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:02.927687 8474 raft_consensus.cc:348] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:02.927769 8470 raft_consensus.cc:374] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.927776 8474 raft_consensus.cc:374] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.927809 8474 raft_consensus.cc:729] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.927839 8470 raft_consensus.cc:729] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.927949 8474 consensus_queue.cc:260] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:02.928021 8470 consensus_queue.cc:260] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:02.928239 8474 ts_tablet_manager.cc:1398] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:02.928457 8474 tablet_bootstrap.cc:492] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:02.928278 8470 ts_tablet_manager.cc:1398] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.928642 8470 tablet_bootstrap.cc:492] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:02.928802 8471 raft_consensus.cc:348] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:02.928910 8471 raft_consensus.cc:374] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.928941 8471 raft_consensus.cc:729] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.929356 8474 tablet_bootstrap.cc:654] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.929080 8471 consensus_queue.cc:260] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:02.929741 8470 tablet_bootstrap.cc:654] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.930694 8473 tablet_bootstrap.cc:492] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:02.930802 8473 ts_tablet_manager.cc:1367] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:02.930485 8474 tablet_bootstrap.cc:492] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:02.931174 8474 ts_tablet_manager.cc:1367] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:02.931654 8474 raft_consensus.cc:348] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:02.931833 8474 raft_consensus.cc:374] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.931910 8474 raft_consensus.cc:729] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.931984 8470 tablet_bootstrap.cc:492] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:02.932085 8470 ts_tablet_manager.cc:1367] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:02.932212 8474 consensus_queue.cc:260] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:02.932556 8470 raft_consensus.cc:348] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:02.932663 8470 raft_consensus.cc:374] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.932700 8470 raft_consensus.cc:729] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.932900 8470 consensus_queue.cc:260] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:02.933296 8470 ts_tablet_manager.cc:1398] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.933364 8474 ts_tablet_manager.cc:1398] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:02.933606 8470 tablet_bootstrap.cc:492] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:02.931330 8471 ts_tablet_manager.cc:1398] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:02.934633 8471 tablet_bootstrap.cc:492] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:02.935220 8473 raft_consensus.cc:348] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.935359 8473 raft_consensus.cc:374] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.935395 8473 raft_consensus.cc:729] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.935832 8470 tablet_bootstrap.cc:654] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.936322 8471 tablet_bootstrap.cc:654] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.937012 8473 consensus_queue.cc:260] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.937803 8473 ts_tablet_manager.cc:1398] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:02.937882 8471 tablet_bootstrap.cc:492] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:02.937964 8471 ts_tablet_manager.cc:1367] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:02.938000 8473 tablet_bootstrap.cc:492] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:02.938482 8471 raft_consensus.cc:348] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.938619 8471 raft_consensus.cc:374] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.938665 8471 raft_consensus.cc:729] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.938798 8471 consensus_queue.cc:260] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.939173 8471 ts_tablet_manager.cc:1398] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.940016 8473 tablet_bootstrap.cc:654] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.941402 8470 tablet_bootstrap.cc:492] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:02.941514 8470 ts_tablet_manager.cc:1367] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:02.942019 8470 raft_consensus.cc:348] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.942153 8470 raft_consensus.cc:374] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.942200 8470 raft_consensus.cc:729] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.942335 8470 consensus_queue.cc:260] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.942715 8470 ts_tablet_manager.cc:1398] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:02.943477 8473 tablet_bootstrap.cc:492] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:02.943572 8473 ts_tablet_manager.cc:1367] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:02.944053 8473 raft_consensus.cc:348] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:02.944172 8473 raft_consensus.cc:374] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.944931 8473 raft_consensus.cc:729] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.945071 8473 consensus_queue.cc:260] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:02.945407 8473 ts_tablet_manager.cc:1398] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:02.963733 8257 raft_consensus.cc:482] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:02.963891 8257 raft_consensus.cc:504] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.964367 8257 leader_election.cc:288] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:02.964846 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b916e89edaf046b99767e5cca531bb8b" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:02.964875 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b916e89edaf046b99767e5cca531bb8b" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:02.964996 6888 raft_consensus.cc:2454] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:02.965029 7149 raft_consensus.cc:2454] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:02.965337 7218 leader_election.cc:302] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:02.965513 8257 raft_consensus.cc:2781] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:02.965600 8257 raft_consensus.cc:482] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:02.965646 8257 raft_consensus.cc:3037] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.966732 8424 raft_consensus.cc:482] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:02.966724 8257 raft_consensus.cc:504] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.966838 8424 raft_consensus.cc:504] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:02.967147 8257 leader_election.cc:288] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:02.967213 8424 leader_election.cc:288] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:02.967599 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccdd597c5b54f41bb846d933a1e9a84" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:02.967612 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccdd597c5b54f41bb846d933a1e9a84" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:02.967721 7278 raft_consensus.cc:2454] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:02.967730 7020 raft_consensus.cc:2454] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:02.967974 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b916e89edaf046b99767e5cca531bb8b" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:02.967979 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b916e89edaf046b99767e5cca531bb8b" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:02.968021 6825 leader_election.cc:302] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:02.968086 6888 raft_consensus.cc:3037] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.968098 7149 raft_consensus.cc:3037] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.968272 8424 raft_consensus.cc:2781] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:02.968328 8424 raft_consensus.cc:482] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:02.968366 8424 raft_consensus.cc:3037] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.968991 7149 raft_consensus.cc:2454] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:02.969116 6888 raft_consensus.cc:2454] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:02.969184 8424 raft_consensus.cc:504] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:02.969341 7218 leader_election.cc:302] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:02.969509 8257 raft_consensus.cc:2781] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:02.969578 8424 leader_election.cc:288] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:02.969668 8257 raft_consensus.cc:686] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:02.969827 8257 consensus_queue.cc:237] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:02.970036 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccdd597c5b54f41bb846d933a1e9a84" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:02.970156 7020 raft_consensus.cc:3037] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.971154 7020 raft_consensus.cc:2454] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:02.971442 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccdd597c5b54f41bb846d933a1e9a84" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:02.971519 6824 leader_election.cc:302] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:02.971549 7278 raft_consensus.cc:3037] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.971676 8424 raft_consensus.cc:2781] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:02.971778 8424 raft_consensus.cc:686] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:02.971902 8424 consensus_queue.cc:237] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:02.972514 7278 raft_consensus.cc:2454] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:02.978073 6739 catalog_manager.cc:5484] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:02.978073 6742 catalog_manager.cc:5484] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:03.221518 8329 raft_consensus.cc:482] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:03.221676 8329 raft_consensus.cc:504] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:03.222110 8329 leader_election.cc:288] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:03.222604 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91af7b042df146e980638f0f79bde295" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:03.222750 6888 raft_consensus.cc:2454] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:03.222738 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91af7b042df146e980638f0f79bde295" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:03.222877 7149 raft_consensus.cc:2454] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:03.223115 6953 leader_election.cc:302] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:03.223304 8329 raft_consensus.cc:2781] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:03.223364 8329 raft_consensus.cc:482] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:03.223392 8329 raft_consensus.cc:3037] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.224299 8329 raft_consensus.cc:504] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:03.224635 8329 leader_election.cc:288] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:03.225013 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91af7b042df146e980638f0f79bde295" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:03.225013 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91af7b042df146e980638f0f79bde295" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:03.225139 7149 raft_consensus.cc:3037] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.225139 6888 raft_consensus.cc:3037] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.226104 6888 raft_consensus.cc:2454] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:03.226104 7149 raft_consensus.cc:2454] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:03.226540 6956 leader_election.cc:302] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:03.226716 8329 raft_consensus.cc:2781] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.226835 8329 raft_consensus.cc:686] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:03.226974 8329 consensus_queue.cc:237] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:03.229754 6739 catalog_manager.cc:5484] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:03.386422 8483 consensus_queue.cc:1035] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.391219 8257 consensus_queue.cc:1035] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:03.417496 8488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:03.417855 8488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:03.421388 8488 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 07:59:03.480926 8424 consensus_queue.cc:1035] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.486096 8192 consensus_queue.cc:1035] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.514811 7149 raft_consensus.cc:1261] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.514870 6888 raft_consensus.cc:1261] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.515331 8328 consensus_queue.cc:1035] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.516813 8328 consensus_queue.cc:1035] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 100.019 ms
time per row: 0.0500095 ms
I20240503 07:59:03.574576 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:03.582278 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2386cd009847d9a1fe7a959f47086c (DEFAULT_TABLE table=rebalance_test_table_9 [id=3e242002464141aaa2f77f2115a129ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.582773 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2386cd009847d9a1fe7a959f47086c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.583236 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2386cd009847d9a1fe7a959f47086c (DEFAULT_TABLE table=rebalance_test_table_9 [id=3e242002464141aaa2f77f2115a129ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.583591 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2386cd009847d9a1fe7a959f47086c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.583236 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcc4c2eeaec4b359108ec64a6e20da8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3e242002464141aaa2f77f2115a129ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.583820 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcc4c2eeaec4b359108ec64a6e20da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.583973 7262 tablet_service.cc:1467] Processing CreateTablet for tablet aa45918bb38c4583ad59524bda2d48ee (DEFAULT_TABLE table=rebalance_test_table_9 [id=3e242002464141aaa2f77f2115a129ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.584228 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa45918bb38c4583ad59524bda2d48ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.586108 8554 tablet_bootstrap.cc:492] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:03.586108 8553 tablet_bootstrap.cc:492] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:03.586582 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2386cd009847d9a1fe7a959f47086c (DEFAULT_TABLE table=rebalance_test_table_9 [id=3e242002464141aaa2f77f2115a129ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.586872 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2386cd009847d9a1fe7a959f47086c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.587136 8553 tablet_bootstrap.cc:654] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.587136 8554 tablet_bootstrap.cc:654] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.589462 6867 tablet_service.cc:1467] Processing CreateTablet for tablet aa45918bb38c4583ad59524bda2d48ee (DEFAULT_TABLE table=rebalance_test_table_9 [id=3e242002464141aaa2f77f2115a129ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.589746 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa45918bb38c4583ad59524bda2d48ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.582278 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcc4c2eeaec4b359108ec64a6e20da8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3e242002464141aaa2f77f2115a129ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.590750 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcc4c2eeaec4b359108ec64a6e20da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.590849 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcc4c2eeaec4b359108ec64a6e20da8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3e242002464141aaa2f77f2115a129ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.591106 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcc4c2eeaec4b359108ec64a6e20da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.594336 8557 tablet_bootstrap.cc:492] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:03.594435 6999 tablet_service.cc:1467] Processing CreateTablet for tablet aa45918bb38c4583ad59524bda2d48ee (DEFAULT_TABLE table=rebalance_test_table_9 [id=3e242002464141aaa2f77f2115a129ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.594728 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa45918bb38c4583ad59524bda2d48ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.595439 8557 tablet_bootstrap.cc:654] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.596596 8557 tablet_bootstrap.cc:492] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:03.596695 8557 ts_tablet_manager.cc:1367] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:03.596812 8553 tablet_bootstrap.cc:492] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:03.596894 8553 ts_tablet_manager.cc:1367] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:03.597227 8557 raft_consensus.cc:348] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:03.597344 8557 raft_consensus.cc:374] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.597309 8553 raft_consensus.cc:348] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:03.597373 8557 raft_consensus.cc:729] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.597404 8553 raft_consensus.cc:374] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.597431 8553 raft_consensus.cc:729] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.597519 8557 consensus_queue.cc:260] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:03.597543 8553 consensus_queue.cc:260] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:03.597815 8557 ts_tablet_manager.cc:1398] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:03.597816 8553 ts_tablet_manager.cc:1398] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:03.597985 8557 tablet_bootstrap.cc:492] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:03.597985 8553 tablet_bootstrap.cc:492] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:03.599010 8553 tablet_bootstrap.cc:654] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.599234 8557 tablet_bootstrap.cc:654] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.600282 8557 tablet_bootstrap.cc:492] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:03.600365 8557 ts_tablet_manager.cc:1367] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:03.600554 8554 tablet_bootstrap.cc:492] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:03.600646 8554 ts_tablet_manager.cc:1367] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:03.600812 8557 raft_consensus.cc:348] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:03.600914 8557 raft_consensus.cc:374] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.600945 8557 raft_consensus.cc:729] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.601074 8557 consensus_queue.cc:260] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:03.601109 8554 raft_consensus.cc:348] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:03.601202 8554 raft_consensus.cc:374] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.601229 8554 raft_consensus.cc:729] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.601370 8557 ts_tablet_manager.cc:1398] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:03.601790 8554 consensus_queue.cc:260] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:03.602773 8554 ts_tablet_manager.cc:1398] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:03.602928 8558 tablet_bootstrap.cc:492] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:03.602990 8554 tablet_bootstrap.cc:492] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:03.603941 8558 tablet_bootstrap.cc:654] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.604009 8553 tablet_bootstrap.cc:492] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:03.604110 8553 ts_tablet_manager.cc:1367] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:03.604569 8553 raft_consensus.cc:348] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:03.604691 8553 raft_consensus.cc:374] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.604732 8553 raft_consensus.cc:729] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.604856 8553 consensus_queue.cc:260] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:03.605180 8553 ts_tablet_manager.cc:1398] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:03.605348 8553 tablet_bootstrap.cc:492] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:03.605801 8554 tablet_bootstrap.cc:654] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.606251 8553 tablet_bootstrap.cc:654] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.607741 8558 tablet_bootstrap.cc:492] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:03.607851 8558 ts_tablet_manager.cc:1367] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:03.608367 8558 raft_consensus.cc:348] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:03.608501 8558 raft_consensus.cc:374] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.608546 8558 raft_consensus.cc:729] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.608700 8558 consensus_queue.cc:260] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:03.609100 8558 ts_tablet_manager.cc:1398] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:03.609781 8553 tablet_bootstrap.cc:492] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:03.609870 8553 ts_tablet_manager.cc:1367] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:03.610527 8553 raft_consensus.cc:348] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:03.610641 8553 raft_consensus.cc:374] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.610672 8553 raft_consensus.cc:729] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.610800 8554 tablet_bootstrap.cc:492] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:03.610776 8553 consensus_queue.cc:260] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:03.610886 8554 ts_tablet_manager.cc:1367] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 07:59:03.611121 8553 ts_tablet_manager.cc:1398] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:03.611353 8554 raft_consensus.cc:348] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:03.611469 8554 raft_consensus.cc:374] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.611501 8554 raft_consensus.cc:729] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.611621 8554 consensus_queue.cc:260] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:03.611908 8554 ts_tablet_manager.cc:1398] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:03.612097 8554 tablet_bootstrap.cc:492] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:03.613137 8554 tablet_bootstrap.cc:654] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.614428 8554 tablet_bootstrap.cc:492] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:03.614527 8554 ts_tablet_manager.cc:1367] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:03.615000 8554 raft_consensus.cc:348] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:03.615123 8554 raft_consensus.cc:374] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.615159 8554 raft_consensus.cc:729] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.615303 8554 consensus_queue.cc:260] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:03.615605 8554 ts_tablet_manager.cc:1398] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:03.687021 8464 raft_consensus.cc:482] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:03.687148 8464 raft_consensus.cc:504] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:03.687584 8464 leader_election.cc:288] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:03.687935 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2386cd009847d9a1fe7a959f47086c" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:03.687989 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2386cd009847d9a1fe7a959f47086c" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:03.688078 7020 raft_consensus.cc:2454] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:03.688162 7278 raft_consensus.cc:2454] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:03.688395 7086 leader_election.cc:302] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:03.688591 8464 raft_consensus.cc:2781] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:03.688657 8464 raft_consensus.cc:482] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:03.688701 8464 raft_consensus.cc:3037] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.689654 8464 raft_consensus.cc:504] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:03.690022 8464 leader_election.cc:288] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:03.690414 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2386cd009847d9a1fe7a959f47086c" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:03.690421 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2386cd009847d9a1fe7a959f47086c" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:03.690542 7278 raft_consensus.cc:3037] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.690548 7020 raft_consensus.cc:3037] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.691375 7020 raft_consensus.cc:2454] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:03.691380 7278 raft_consensus.cc:2454] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:03.691690 7086 leader_election.cc:302] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:03.691869 8464 raft_consensus.cc:2781] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.691952 8464 raft_consensus.cc:686] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:03.692130 8464 consensus_queue.cc:237] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:03.694986 6741 catalog_manager.cc:5484] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:03.744197 8531 raft_consensus.cc:482] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:03.744343 8531 raft_consensus.cc:504] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:03.744722 8531 leader_election.cc:288] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:03.745160 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc4c2eeaec4b359108ec64a6e20da8" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:03.745188 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc4c2eeaec4b359108ec64a6e20da8" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:03.745337 7151 raft_consensus.cc:2454] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:03.745361 7020 raft_consensus.cc:2454] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:03.745683 7218 leader_election.cc:302] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:03.745911 8531 raft_consensus.cc:2781] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:03.745986 8531 raft_consensus.cc:482] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:03.746023 8531 raft_consensus.cc:3037] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.746995 8531 raft_consensus.cc:504] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:03.747337 8531 leader_election.cc:288] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:03.747710 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc4c2eeaec4b359108ec64a6e20da8" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:03.747730 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc4c2eeaec4b359108ec64a6e20da8" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:03.747845 7020 raft_consensus.cc:3037] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.747844 7151 raft_consensus.cc:3037] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.748839 7151 raft_consensus.cc:2454] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:03.748839 7020 raft_consensus.cc:2454] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:03.749184 7218 leader_election.cc:302] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:03.749365 8531 raft_consensus.cc:2781] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.749449 8531 raft_consensus.cc:686] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:03.749593 8531 consensus_queue.cc:237] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:03.752157 6741 catalog_manager.cc:5484] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:03.753149 8531 raft_consensus.cc:482] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:03.753255 8531 raft_consensus.cc:504] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:03.753669 8531 leader_election.cc:288] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:03.754024 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa45918bb38c4583ad59524bda2d48ee" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:03.754206 6888 raft_consensus.cc:2454] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:03.754563 7215 leader_election.cc:302] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:03.754736 8531 raft_consensus.cc:2781] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:03.754757 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa45918bb38c4583ad59524bda2d48ee" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:03.754802 8531 raft_consensus.cc:482] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:03.754837 8531 raft_consensus.cc:3037] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.754879 7020 raft_consensus.cc:2454] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:03.755725 8531 raft_consensus.cc:504] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:03.756058 8531 leader_election.cc:288] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:03.756443 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa45918bb38c4583ad59524bda2d48ee" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:03.756443 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa45918bb38c4583ad59524bda2d48ee" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:03.756554 7020 raft_consensus.cc:3037] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.756554 6888 raft_consensus.cc:3037] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.757534 7020 raft_consensus.cc:2454] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:03.757534 6888 raft_consensus.cc:2454] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:03.757841 7215 leader_election.cc:302] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:03.758011 8531 raft_consensus.cc:2781] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.758106 8531 raft_consensus.cc:686] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:03.758250 8531 consensus_queue.cc:237] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:03.760092 6746 catalog_manager.cc:5484] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
W20240503 07:59:03.915107 8569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:03.915442 8569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:03.919078 8569 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 07:59:04.012025 7020 raft_consensus.cc:1261] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.012468 8531 consensus_queue.cc:1035] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.015368 7020 raft_consensus.cc:1261] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.015802 8531 consensus_queue.cc:1035] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.016335 7151 raft_consensus.cc:1261] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.017128 8531 consensus_queue.cc:1035] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.018965 7278 raft_consensus.cc:1261] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.019086 7018 raft_consensus.cc:1261] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.019423 8456 consensus_queue.cc:1035] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.020269 8456 consensus_queue.cc:1035] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.020704 6888 raft_consensus.cc:1261] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.021265 8531 consensus_queue.cc:1035] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 108.721 ms
time per row: 0.0543607 ms
I20240503 07:59:04.082099 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:04.088562 7129 tablet_service.cc:1467] Processing CreateTablet for tablet c86d285233c84f628d2ce4c531e3aef4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.088923 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 20d8ad35129042a093035013305e42b5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.088938 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86d285233c84f628d2ce4c531e3aef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.089381 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbcf7ab42a1463582e96927d7f16b9a (DEFAULT_TABLE table=rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.089399 6867 tablet_service.cc:1467] Processing CreateTablet for tablet c86d285233c84f628d2ce4c531e3aef4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.089596 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbcf7ab42a1463582e96927d7f16b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.089607 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbcf7ab42a1463582e96927d7f16b9a (DEFAULT_TABLE table=rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.089694 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86d285233c84f628d2ce4c531e3aef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.089795 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbcf7ab42a1463582e96927d7f16b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.091035 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 20d8ad35129042a093035013305e42b5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.091285 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d8ad35129042a093035013305e42b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.091300 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbcf7ab42a1463582e96927d7f16b9a (DEFAULT_TABLE table=rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.091563 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbcf7ab42a1463582e96927d7f16b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.092001 8557 tablet_bootstrap.cc:492] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:04.092941 8557 tablet_bootstrap.cc:654] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.093613 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d8ad35129042a093035013305e42b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.093672 8558 tablet_bootstrap.cc:492] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:04.093923 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 20d8ad35129042a093035013305e42b5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.094249 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d8ad35129042a093035013305e42b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.094642 8558 tablet_bootstrap.cc:654] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.095288 8553 tablet_bootstrap.cc:492] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:04.095849 8558 tablet_bootstrap.cc:492] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:04.095944 8558 ts_tablet_manager.cc:1367] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:04.096261 8553 tablet_bootstrap.cc:654] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.096391 8558 raft_consensus.cc:348] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:04.096534 8558 raft_consensus.cc:374] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.096570 8558 raft_consensus.cc:729] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.096697 8558 consensus_queue.cc:260] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:04.096926 8554 tablet_bootstrap.cc:492] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:04.097700 8558 ts_tablet_manager.cc:1398] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:04.097856 8554 tablet_bootstrap.cc:654] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.098698 6999 tablet_service.cc:1467] Processing CreateTablet for tablet c86d285233c84f628d2ce4c531e3aef4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.098819 8558 tablet_bootstrap.cc:492] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:04.098986 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86d285233c84f628d2ce4c531e3aef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.099678 8558 tablet_bootstrap.cc:654] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.101059 8558 tablet_bootstrap.cc:492] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:04.101145 8558 ts_tablet_manager.cc:1367] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:04.101230 8554 tablet_bootstrap.cc:492] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:04.101338 8554 ts_tablet_manager.cc:1367] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:04.101614 8558 raft_consensus.cc:348] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.101739 8558 raft_consensus.cc:374] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.101771 8558 raft_consensus.cc:729] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.101783 8554 raft_consensus.cc:348] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:04.101907 8554 raft_consensus.cc:374] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.101892 8558 consensus_queue.cc:260] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.101939 8554 raft_consensus.cc:729] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.101955 8553 tablet_bootstrap.cc:492] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:04.102035 8553 ts_tablet_manager.cc:1367] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:04.102077 8554 consensus_queue.cc:260] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:04.102201 8558 ts_tablet_manager.cc:1398] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.102382 8554 ts_tablet_manager.cc:1398] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.102398 8558 tablet_bootstrap.cc:492] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:04.102537 8553 raft_consensus.cc:348] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.102644 8553 raft_consensus.cc:374] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.102679 8553 raft_consensus.cc:729] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.102807 8553 consensus_queue.cc:260] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.103667 8553 ts_tablet_manager.cc:1398] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:04.103859 8553 tablet_bootstrap.cc:492] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:04.104030 8558 tablet_bootstrap.cc:654] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.104745 8553 tablet_bootstrap.cc:654] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.105901 8557 tablet_bootstrap.cc:492] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:04.106015 8557 ts_tablet_manager.cc:1367] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20240503 07:59:04.106295 8558 tablet_bootstrap.cc:492] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:04.106473 8558 ts_tablet_manager.cc:1367] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:04.106555 8557 raft_consensus.cc:348] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.106683 8557 raft_consensus.cc:374] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.106738 8553 tablet_bootstrap.cc:492] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:04.106850 8553 ts_tablet_manager.cc:1367] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:04.106716 8557 raft_consensus.cc:729] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.107013 8557 consensus_queue.cc:260] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.107298 8553 raft_consensus.cc:348] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.107347 8557 ts_tablet_manager.cc:1398] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:04.107409 8553 raft_consensus.cc:374] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.107446 8553 raft_consensus.cc:729] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.107563 8557 tablet_bootstrap.cc:492] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:04.107558 8553 consensus_queue.cc:260] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.107743 8558 raft_consensus.cc:348] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.107864 8553 ts_tablet_manager.cc:1398] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:04.107897 8558 raft_consensus.cc:374] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.108016 8558 raft_consensus.cc:729] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.108250 8558 consensus_queue.cc:260] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.108665 8558 ts_tablet_manager.cc:1398] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:04.108958 8557 tablet_bootstrap.cc:654] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.110388 8557 tablet_bootstrap.cc:492] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:04.110486 8557 ts_tablet_manager.cc:1367] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:04.111092 8557 raft_consensus.cc:348] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:04.111212 8557 raft_consensus.cc:374] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.111244 8557 raft_consensus.cc:729] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.111891 8557 consensus_queue.cc:260] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:04.112229 8557 ts_tablet_manager.cc:1398] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:04.112421 8557 tablet_bootstrap.cc:492] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:04.113426 8557 tablet_bootstrap.cc:654] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.114672 8557 tablet_bootstrap.cc:492] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:04.114761 8557 ts_tablet_manager.cc:1367] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:04.115203 8557 raft_consensus.cc:348] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.115321 8557 raft_consensus.cc:374] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.115365 8557 raft_consensus.cc:729] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.115495 8557 consensus_queue.cc:260] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.115799 8557 ts_tablet_manager.cc:1398] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:04.180241 8608 raft_consensus.cc:482] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:04.180403 8608 raft_consensus.cc:504] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:04.180904 8608 leader_election.cc:288] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:04.181380 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d8ad35129042a093035013305e42b5" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:04.181381 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d8ad35129042a093035013305e42b5" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:04.181540 6888 raft_consensus.cc:2454] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:04.181562 7279 raft_consensus.cc:2454] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:04.182006 7084 leader_election.cc:302] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:04.182251 8608 raft_consensus.cc:2781] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:04.182359 8608 raft_consensus.cc:482] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:04.182395 8608 raft_consensus.cc:3037] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.183889 8608 raft_consensus.cc:504] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:04.184285 8608 leader_election.cc:288] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:04.184598 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d8ad35129042a093035013305e42b5" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:04.184629 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d8ad35129042a093035013305e42b5" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:04.184742 7279 raft_consensus.cc:3037] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.184758 6888 raft_consensus.cc:3037] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.185675 6888 raft_consensus.cc:2454] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:04.185688 7279 raft_consensus.cc:2454] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:04.186025 7084 leader_election.cc:302] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:04.186260 8608 raft_consensus.cc:2781] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:04.186393 8608 raft_consensus.cc:686] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:04.186532 8608 consensus_queue.cc:237] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:04.188438 6741 catalog_manager.cc:5484] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:04.230206 8192 raft_consensus.cc:482] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:04.230366 8192 raft_consensus.cc:504] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.230818 8192 leader_election.cc:288] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:04.231334 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cbcf7ab42a1463582e96927d7f16b9a" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:04.231329 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cbcf7ab42a1463582e96927d7f16b9a" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:04.231477 7151 raft_consensus.cc:2454] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:04.231484 7020 raft_consensus.cc:2454] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:04.231786 6824 leader_election.cc:302] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:04.231963 8192 raft_consensus.cc:2781] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:04.232023 8192 raft_consensus.cc:482] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:04.232054 8192 raft_consensus.cc:3037] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.232971 8192 raft_consensus.cc:504] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.233338 8192 leader_election.cc:288] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:04.233729 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cbcf7ab42a1463582e96927d7f16b9a" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:04.233729 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cbcf7ab42a1463582e96927d7f16b9a" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:04.233856 7020 raft_consensus.cc:3037] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.233886 7151 raft_consensus.cc:3037] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.234831 7151 raft_consensus.cc:2454] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:04.234841 7020 raft_consensus.cc:2454] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:04.235092 8328 raft_consensus.cc:482] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:04.235172 8328 raft_consensus.cc:504] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.235172 6825 leader_election.cc:302] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:04.235334 8192 raft_consensus.cc:2781] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:04.235417 8192 raft_consensus.cc:686] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:04.235553 8328 leader_election.cc:288] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:04.235592 8192 consensus_queue.cc:237] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.236063 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86d285233c84f628d2ce4c531e3aef4" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:04.236063 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86d285233c84f628d2ce4c531e3aef4" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:04.236183 7151 raft_consensus.cc:2454] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:04.236183 6888 raft_consensus.cc:2454] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:04.236459 6953 leader_election.cc:302] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:04.236617 8328 raft_consensus.cc:2781] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:04.236677 8328 raft_consensus.cc:482] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:04.236721 8328 raft_consensus.cc:3037] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.237658 8328 raft_consensus.cc:504] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.237673 6741 catalog_manager.cc:5484] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:04.238032 8328 leader_election.cc:288] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:04.239112 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86d285233c84f628d2ce4c531e3aef4" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:04.239235 6889 raft_consensus.cc:3037] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.240057 6889 raft_consensus.cc:2454] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:04.240377 6953 leader_election.cc:302] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:04.240594 8328 raft_consensus.cc:2781] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:04.240741 8328 raft_consensus.cc:686] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:04.240749 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86d285233c84f628d2ce4c531e3aef4" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:04.240881 7151 raft_consensus.cc:3037] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.241134 8328 consensus_queue.cc:237] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.241892 7151 raft_consensus.cc:2454] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:04.243149 6741 catalog_manager.cc:5484] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
W20240503 07:59:04.384881 8621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:04.385270 8621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:04.390159 8621 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240503 07:59:04.488104 7020 raft_consensus.cc:1261] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.488498 7151 raft_consensus.cc:1261] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.489118 8589 consensus_queue.cc:1035] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.489496 8589 consensus_queue.cc:1035] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.490315 7278 raft_consensus.cc:1261] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.490334 6889 raft_consensus.cc:1261] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.490768 8608 consensus_queue.cc:1035] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.491358 8608 consensus_queue.cc:1035] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.492427 6888 raft_consensus.cc:1261] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.493341 7151 raft_consensus.cc:1261] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.493845 8328 consensus_queue.cc:1035] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.495048 8329 consensus_queue.cc:1035] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.077 ms
time per row: 0.0595387 ms
I20240503 07:59:04.562206 6746 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:04.568485 7262 tablet_service.cc:1467] Processing CreateTablet for tablet f365955974634bc58e8a0da0fc2563d7 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9caa58273688477ab18820b0590675ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.568879 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f365955974634bc58e8a0da0fc2563d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.568984 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 6e63db30c0454722b6a794a9259a67c0 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9caa58273688477ab18820b0590675ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.569244 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e63db30c0454722b6a794a9259a67c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.569948 6869 tablet_service.cc:1467] Processing CreateTablet for tablet f365955974634bc58e8a0da0fc2563d7 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9caa58273688477ab18820b0590675ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.570233 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f365955974634bc58e8a0da0fc2563d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.570634 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 6e63db30c0454722b6a794a9259a67c0 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9caa58273688477ab18820b0590675ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.570858 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e63db30c0454722b6a794a9259a67c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.571395 8554 tablet_bootstrap.cc:492] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:04.572256 8554 tablet_bootstrap.cc:654] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.572415 8558 tablet_bootstrap.cc:492] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:04.573196 8558 tablet_bootstrap.cc:654] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.574211 8558 tablet_bootstrap.cc:492] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:04.574308 8558 ts_tablet_manager.cc:1367] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:04.574594 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 6e63db30c0454722b6a794a9259a67c0 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9caa58273688477ab18820b0590675ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.574811 8558 raft_consensus.cc:348] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.574923 8558 raft_consensus.cc:374] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.574859 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 921bd35f61ed4465aef45eee3383c9b3 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9caa58273688477ab18820b0590675ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.574959 8558 raft_consensus.cc:729] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.574867 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e63db30c0454722b6a794a9259a67c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.575083 8558 consensus_queue.cc:260] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.575361 8558 ts_tablet_manager.cc:1398] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:04.575531 8558 tablet_bootstrap.cc:492] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:04.576234 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921bd35f61ed4465aef45eee3383c9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.576536 8558 tablet_bootstrap.cc:654] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.577165 8557 tablet_bootstrap.cc:492] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:04.578029 8557 tablet_bootstrap.cc:654] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.578027 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 921bd35f61ed4465aef45eee3383c9b3 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9caa58273688477ab18820b0590675ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.578572 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921bd35f61ed4465aef45eee3383c9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.578908 6999 tablet_service.cc:1467] Processing CreateTablet for tablet f365955974634bc58e8a0da0fc2563d7 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9caa58273688477ab18820b0590675ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.579186 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f365955974634bc58e8a0da0fc2563d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.580125 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 921bd35f61ed4465aef45eee3383c9b3 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9caa58273688477ab18820b0590675ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.580375 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921bd35f61ed4465aef45eee3383c9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.581457 8553 tablet_bootstrap.cc:492] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:04.581631 8557 tablet_bootstrap.cc:492] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:04.581724 8557 ts_tablet_manager.cc:1367] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:04.582290 8553 tablet_bootstrap.cc:654] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.582196 8557 raft_consensus.cc:348] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:04.582365 8557 raft_consensus.cc:374] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.582404 8557 raft_consensus.cc:729] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.582528 8557 consensus_queue.cc:260] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:04.582818 8557 ts_tablet_manager.cc:1398] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.582983 8557 tablet_bootstrap.cc:492] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:04.583798 8553 tablet_bootstrap.cc:492] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:04.583866 8557 tablet_bootstrap.cc:654] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.583899 8553 ts_tablet_manager.cc:1367] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:04.584338 8553 raft_consensus.cc:348] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.584465 8553 raft_consensus.cc:374] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.584501 8553 raft_consensus.cc:729] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.584622 8553 consensus_queue.cc:260] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.584934 8553 ts_tablet_manager.cc:1398] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.585114 8553 tablet_bootstrap.cc:492] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:04.585990 8553 tablet_bootstrap.cc:654] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.588405 8553 tablet_bootstrap.cc:492] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:04.588557 8553 ts_tablet_manager.cc:1367] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:04.588613 8554 tablet_bootstrap.cc:492] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:04.588721 8554 ts_tablet_manager.cc:1367] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.000s
I20240503 07:59:04.589124 8554 raft_consensus.cc:348] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:04.589239 8554 raft_consensus.cc:374] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.589272 8554 raft_consensus.cc:729] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.589391 8554 consensus_queue.cc:260] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:04.589470 8553 raft_consensus.cc:348] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:04.589569 8553 raft_consensus.cc:374] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.589596 8553 raft_consensus.cc:729] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.589682 8554 ts_tablet_manager.cc:1398] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.589730 8553 consensus_queue.cc:260] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:04.589859 8554 tablet_bootstrap.cc:492] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:04.590054 8553 ts_tablet_manager.cc:1398] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:04.590724 8554 tablet_bootstrap.cc:654] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.591382 8557 tablet_bootstrap.cc:492] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:04.591481 8557 ts_tablet_manager.cc:1367] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:04.591926 8557 raft_consensus.cc:348] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:04.592034 8557 raft_consensus.cc:374] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.592065 8557 raft_consensus.cc:729] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.592203 8557 consensus_queue.cc:260] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:04.592551 8557 ts_tablet_manager.cc:1398] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.592952 8558 tablet_bootstrap.cc:492] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:04.593075 8558 ts_tablet_manager.cc:1367] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.001s
I20240503 07:59:04.593559 8558 raft_consensus.cc:348] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:04.593676 8558 raft_consensus.cc:374] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.593715 8558 raft_consensus.cc:729] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.593842 8558 consensus_queue.cc:260] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:04.594082 8554 tablet_bootstrap.cc:492] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:04.594111 8558 ts_tablet_manager.cc:1398] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:04.594214 8554 ts_tablet_manager.cc:1367] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:04.595728 8554 raft_consensus.cc:348] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.595834 8554 raft_consensus.cc:374] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.595862 8554 raft_consensus.cc:729] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.596005 8554 consensus_queue.cc:260] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.596395 8554 ts_tablet_manager.cc:1398] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:04.596611 8554 tablet_bootstrap.cc:492] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:04.598194 8605 raft_consensus.cc:482] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:04.598281 8605 raft_consensus.cc:504] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:04.598688 8605 leader_election.cc:288] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:04.598716 8554 tablet_bootstrap.cc:654] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.599103 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e63db30c0454722b6a794a9259a67c0" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:04.599131 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e63db30c0454722b6a794a9259a67c0" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:04.599249 7149 raft_consensus.cc:2454] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:04.599248 6889 raft_consensus.cc:2454] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:04.599565 7215 leader_election.cc:302] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:04.599794 8605 raft_consensus.cc:2781] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:04.599840 8605 raft_consensus.cc:482] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:04.599874 8605 raft_consensus.cc:3037] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.600299 8554 tablet_bootstrap.cc:492] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:04.600379 8554 ts_tablet_manager.cc:1367] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 07:59:04.600857 8605 raft_consensus.cc:504] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:04.601209 8605 leader_election.cc:288] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:04.601276 8554 raft_consensus.cc:348] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:04.601368 8554 raft_consensus.cc:374] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.601395 8554 raft_consensus.cc:729] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.601533 8554 consensus_queue.cc:260] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:04.601780 8554 ts_tablet_manager.cc:1398] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.602488 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e63db30c0454722b6a794a9259a67c0" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:04.602619 6889 raft_consensus.cc:3037] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.602876 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e63db30c0454722b6a794a9259a67c0" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:04.602986 7149 raft_consensus.cc:3037] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.603690 6889 raft_consensus.cc:2454] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:04.603909 7149 raft_consensus.cc:2454] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:04.604076 7215 leader_election.cc:302] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:04.604525 8605 raft_consensus.cc:2781] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:04.604621 8605 raft_consensus.cc:686] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:04.604764 8605 consensus_queue.cc:237] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:04.607360 6742 catalog_manager.cc:5484] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:04.649906 8456 raft_consensus.cc:482] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:04.650068 8456 raft_consensus.cc:504] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:04.650599 8456 leader_election.cc:288] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:04.651057 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921bd35f61ed4465aef45eee3383c9b3" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:04.651058 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921bd35f61ed4465aef45eee3383c9b3" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:04.651206 7020 raft_consensus.cc:2454] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:04.651213 7278 raft_consensus.cc:2454] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:04.651520 7086 leader_election.cc:302] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:04.651687 8456 raft_consensus.cc:2781] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:04.651748 8456 raft_consensus.cc:482] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:04.651796 8456 raft_consensus.cc:3037] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.652767 8456 raft_consensus.cc:504] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:04.653123 8456 leader_election.cc:288] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:04.653492 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921bd35f61ed4465aef45eee3383c9b3" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:04.653492 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921bd35f61ed4465aef45eee3383c9b3" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:04.653625 7278 raft_consensus.cc:3037] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.653625 7018 raft_consensus.cc:3037] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.654706 7278 raft_consensus.cc:2454] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:04.654706 7018 raft_consensus.cc:2454] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:04.655033 7086 leader_election.cc:302] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:04.655221 8456 raft_consensus.cc:2781] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:04.655308 8456 raft_consensus.cc:686] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:04.655454 8456 consensus_queue.cc:237] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:04.657341 6742 catalog_manager.cc:5484] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:04.685312 8589 raft_consensus.cc:482] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:04.685489 8589 raft_consensus.cc:504] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.685971 8589 leader_election.cc:288] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:04.686527 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f365955974634bc58e8a0da0fc2563d7" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:04.686529 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f365955974634bc58e8a0da0fc2563d7" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:04.686681 7018 raft_consensus.cc:2454] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:04.686686 7278 raft_consensus.cc:2454] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:04.687022 6824 leader_election.cc:302] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:04.687212 8589 raft_consensus.cc:2781] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:04.687294 8589 raft_consensus.cc:482] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:04.687322 8589 raft_consensus.cc:3037] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.688247 8589 raft_consensus.cc:504] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.688649 8589 leader_election.cc:288] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:04.689005 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f365955974634bc58e8a0da0fc2563d7" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:04.689122 7278 raft_consensus.cc:3037] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.689081 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f365955974634bc58e8a0da0fc2563d7" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:04.689213 7018 raft_consensus.cc:3037] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.689929 7278 raft_consensus.cc:2454] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:04.690037 7018 raft_consensus.cc:2454] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:04.690245 6825 leader_election.cc:302] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:04.690447 8589 raft_consensus.cc:2781] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:04.690538 8589 raft_consensus.cc:686] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:04.690711 8589 consensus_queue.cc:237] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:04.692888 6742 catalog_manager.cc:5484] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
W20240503 07:59:04.838230 8679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:04.838577 8679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:04.842322 8679 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240503 07:59:04.933200 7278 raft_consensus.cc:1261] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.933200 7018 raft_consensus.cc:1261] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.933367 7149 raft_consensus.cc:1261] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.933674 8589 consensus_queue.cc:1035] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.934218 8605 consensus_queue.cc:1035] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.934387 8589 consensus_queue.cc:1035] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.941658 6889 raft_consensus.cc:1261] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.946779 8605 consensus_queue.cc:1035] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.948237 7020 raft_consensus.cc:1261] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.948324 7278 raft_consensus.cc:1261] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.948701 8456 consensus_queue.cc:1035] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.949100 8608 consensus_queue.cc:1035] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.963 ms
time per row: 0.0599816 ms
I20240503 07:59:05.016880 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:05.023550 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 85bf549243424602b6615773cf8bee33 (DEFAULT_TABLE table=rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.023955 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bf549243424602b6615773cf8bee33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.024099 7131 tablet_service.cc:1467] Processing CreateTablet for tablet e32f28262dfb4c738bf8e939902ef181 (DEFAULT_TABLE table=rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.024412 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32f28262dfb4c738bf8e939902ef181. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.025710 6869 tablet_service.cc:1467] Processing CreateTablet for tablet e32f28262dfb4c738bf8e939902ef181 (DEFAULT_TABLE table=rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.026024 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32f28262dfb4c738bf8e939902ef181. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.026257 8553 tablet_bootstrap.cc:492] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:05.026685 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 85bf549243424602b6615773cf8bee33 (DEFAULT_TABLE table=rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.026983 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bf549243424602b6615773cf8bee33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.027343 8553 tablet_bootstrap.cc:654] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.028115 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 96c14f4b2a054ea98bde16bc8d4422ac (DEFAULT_TABLE table=rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.028383 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c14f4b2a054ea98bde16bc8d4422ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.028400 8558 tablet_bootstrap.cc:492] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:05.029105 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 96c14f4b2a054ea98bde16bc8d4422ac (DEFAULT_TABLE table=rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.029317 8558 tablet_bootstrap.cc:654] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.029376 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c14f4b2a054ea98bde16bc8d4422ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.029426 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 85bf549243424602b6615773cf8bee33 (DEFAULT_TABLE table=rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.029657 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bf549243424602b6615773cf8bee33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.030633 8558 tablet_bootstrap.cc:492] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:05.030603 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 96c14f4b2a054ea98bde16bc8d4422ac (DEFAULT_TABLE table=rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.030709 8558 ts_tablet_manager.cc:1367] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:05.030829 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c14f4b2a054ea98bde16bc8d4422ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.031203 8558 raft_consensus.cc:348] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.031304 8558 raft_consensus.cc:374] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.031330 8558 raft_consensus.cc:729] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.031446 8558 consensus_queue.cc:260] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.031718 8558 ts_tablet_manager.cc:1398] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.032902 8554 tablet_bootstrap.cc:492] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:05.032946 8558 tablet_bootstrap.cc:492] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:05.033813 8554 tablet_bootstrap.cc:654] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.033979 8558 tablet_bootstrap.cc:654] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.034500 8557 tablet_bootstrap.cc:492] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:05.034992 8554 tablet_bootstrap.cc:492] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:05.035079 8554 ts_tablet_manager.cc:1367] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:05.035454 8558 tablet_bootstrap.cc:492] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:05.035516 8558 ts_tablet_manager.cc:1367] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:05.035499 8557 tablet_bootstrap.cc:654] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.035842 8558 raft_consensus.cc:348] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.035835 8554 raft_consensus.cc:348] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.035933 8558 raft_consensus.cc:374] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.035951 8558 raft_consensus.cc:729] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.035953 8554 raft_consensus.cc:374] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.035984 8554 raft_consensus.cc:729] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.036058 8558 consensus_queue.cc:260] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.036219 8554 consensus_queue.cc:260] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.036922 8557 tablet_bootstrap.cc:492] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:05.037015 8557 ts_tablet_manager.cc:1367] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:05.037483 6998 tablet_service.cc:1467] Processing CreateTablet for tablet e32f28262dfb4c738bf8e939902ef181 (DEFAULT_TABLE table=rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.037472 8557 raft_consensus.cc:348] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.037593 8557 raft_consensus.cc:374] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.037621 8557 raft_consensus.cc:729] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.037776 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32f28262dfb4c738bf8e939902ef181. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.037775 8557 consensus_queue.cc:260] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.038036 8557 ts_tablet_manager.cc:1398] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.039306 8554 ts_tablet_manager.cc:1398] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:05.039556 8554 tablet_bootstrap.cc:492] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:05.039562 8553 tablet_bootstrap.cc:492] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:05.039664 8553 ts_tablet_manager.cc:1367] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:05.040136 8553 raft_consensus.cc:348] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.040259 8553 raft_consensus.cc:374] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.040292 8553 raft_consensus.cc:729] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.040416 8553 consensus_queue.cc:260] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.040710 8553 ts_tablet_manager.cc:1398] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.040896 8553 tablet_bootstrap.cc:492] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:05.041890 8553 tablet_bootstrap.cc:654] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.042789 8554 tablet_bootstrap.cc:654] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.043011 8557 tablet_bootstrap.cc:492] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:05.043105 8558 ts_tablet_manager.cc:1398] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:05.044138 8557 tablet_bootstrap.cc:654] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.045121 8558 tablet_bootstrap.cc:492] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:05.045255 8554 tablet_bootstrap.cc:492] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:05.045341 8554 ts_tablet_manager.cc:1367] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:05.045843 8554 raft_consensus.cc:348] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.045980 8554 raft_consensus.cc:374] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.046022 8554 raft_consensus.cc:729] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.046173 8554 consensus_queue.cc:260] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.046314 8558 tablet_bootstrap.cc:654] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.047099 8554 ts_tablet_manager.cc:1398] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:05.047289 8557 tablet_bootstrap.cc:492] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:05.047289 8553 tablet_bootstrap.cc:492] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:05.047390 8553 ts_tablet_manager.cc:1367] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:05.047390 8557 ts_tablet_manager.cc:1367] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:05.048012 8557 raft_consensus.cc:348] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.047988 8553 raft_consensus.cc:348] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.048130 8553 raft_consensus.cc:374] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.048161 8557 raft_consensus.cc:374] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.048223 8557 raft_consensus.cc:729] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.048374 8557 consensus_queue.cc:260] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.048174 8553 raft_consensus.cc:729] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.048774 8557 ts_tablet_manager.cc:1398] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.048969 8553 consensus_queue.cc:260] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.049296 8558 tablet_bootstrap.cc:492] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:05.049407 8558 ts_tablet_manager.cc:1367] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:05.049918 8558 raft_consensus.cc:348] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.050029 8558 raft_consensus.cc:374] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.050056 8558 raft_consensus.cc:729] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.050177 8558 consensus_queue.cc:260] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.050482 8558 ts_tablet_manager.cc:1398] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.051862 8553 ts_tablet_manager.cc:1398] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:05.077311 8329 raft_consensus.cc:482] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.077440 8329 raft_consensus.cc:504] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.077860 8329 leader_election.cc:288] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:05.078336 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bf549243424602b6615773cf8bee33" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:05.078362 6887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bf549243424602b6615773cf8bee33" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:05.078534 6887 raft_consensus.cc:2454] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:05.078534 7278 raft_consensus.cc:2454] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:05.078863 6953 leader_election.cc:302] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:05.079067 8328 raft_consensus.cc:2781] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.079149 8328 raft_consensus.cc:482] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.079188 8328 raft_consensus.cc:3037] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.080336 8328 raft_consensus.cc:504] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.080720 8328 leader_election.cc:288] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:05.081024 6887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bf549243424602b6615773cf8bee33" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:05.081156 6887 raft_consensus.cc:3037] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.081147 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bf549243424602b6615773cf8bee33" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:05.081311 7278 raft_consensus.cc:3037] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.082103 6887 raft_consensus.cc:2454] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:05.082105 7278 raft_consensus.cc:2454] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:05.082453 6953 leader_election.cc:302] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:05.082672 8328 raft_consensus.cc:2781] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.082764 8328 raft_consensus.cc:686] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:05.082924 8328 consensus_queue.cc:237] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.084867 6738 catalog_manager.cc:5484] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:05.193113 8456 raft_consensus.cc:482] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.193281 8456 raft_consensus.cc:504] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.193766 8456 leader_election.cc:288] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:05.194298 6887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32f28262dfb4c738bf8e939902ef181" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:05.194298 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32f28262dfb4c738bf8e939902ef181" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:05.194491 6887 raft_consensus.cc:2454] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:05.194491 7020 raft_consensus.cc:2454] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:05.194869 7084 leader_election.cc:302] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:05.195053 8456 raft_consensus.cc:2781] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.195122 8456 raft_consensus.cc:482] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.195168 8456 raft_consensus.cc:3037] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.196125 8456 raft_consensus.cc:504] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.196516 8456 leader_election.cc:288] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:05.196902 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32f28262dfb4c738bf8e939902ef181" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:05.197039 7020 raft_consensus.cc:3037] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.196990 6887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32f28262dfb4c738bf8e939902ef181" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:05.197136 6887 raft_consensus.cc:3037] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.197903 7020 raft_consensus.cc:2454] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:05.198251 7086 leader_election.cc:302] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:05.198300 6887 raft_consensus.cc:2454] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:05.198489 8456 raft_consensus.cc:2781] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.198685 8456 raft_consensus.cc:686] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:05.198876 8456 consensus_queue.cc:237] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.200969 6738 catalog_manager.cc:5484] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:05.208284 8456 raft_consensus.cc:482] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.208433 8456 raft_consensus.cc:504] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.208880 8456 leader_election.cc:288] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:05.209405 6887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c14f4b2a054ea98bde16bc8d4422ac" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:05.209401 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c14f4b2a054ea98bde16bc8d4422ac" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:05.209569 6887 raft_consensus.cc:2454] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:05.209569 7278 raft_consensus.cc:2454] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:05.209936 7087 leader_election.cc:302] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:05.210122 8456 raft_consensus.cc:2781] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.210186 8456 raft_consensus.cc:482] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.210213 8456 raft_consensus.cc:3037] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.211211 8456 raft_consensus.cc:504] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.211652 8456 leader_election.cc:288] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:05.212039 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c14f4b2a054ea98bde16bc8d4422ac" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:05.212193 7278 raft_consensus.cc:3037] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.213057 7278 raft_consensus.cc:2454] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:05.213347 6887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c14f4b2a054ea98bde16bc8d4422ac" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:05.213393 7087 leader_election.cc:302] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:05.213491 6887 raft_consensus.cc:3037] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.213563 8456 raft_consensus.cc:2781] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.213699 8456 raft_consensus.cc:686] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:05.213855 8456 consensus_queue.cc:237] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:05.215237 6887 raft_consensus.cc:2454] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:05.216053 6738 catalog_manager.cc:5484] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
W20240503 07:59:05.361449 8730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:05.361747 8730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:05.365132 8730 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240503 07:59:05.486508 7020 raft_consensus.cc:1261] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.486632 7278 raft_consensus.cc:1261] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.486708 6889 raft_consensus.cc:1261] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.486968 8456 consensus_queue.cc:1035] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.489022 8329 consensus_queue.cc:1035] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.490409 8464 consensus_queue.cc:1035] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.491815 7279 raft_consensus.cc:1261] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.492211 8464 consensus_queue.cc:1035] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.492607 6887 raft_consensus.cc:1261] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.492607 6888 raft_consensus.cc:1261] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.495074 8608 consensus_queue.cc:1035] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.506521 8329 consensus_queue.cc:1035] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.511296 8761 mvcc.cc:205] Tried to move back new op lower bound from 7023506003927592960 to 7023506002797813760. Current Snapshot: MvccSnapshot[applied={T|T < 7023506003927592960}]
I20240503 07:59:05.522665 8766 mvcc.cc:205] Tried to move back new op lower bound from 7023506003904503808 to 7023506002261037056. Current Snapshot: MvccSnapshot[applied={T|T < 7023506003904503808}]
INSERT report
rows total: 2000
time total: 125.332 ms
time per row: 0.0626662 ms
I20240503 07:59:05.552013 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:05.559077 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 5365969434f142b8a5757a9a73bc109a (DEFAULT_TABLE table=rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.559463 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5365969434f142b8a5757a9a73bc109a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.559626 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 (DEFAULT_TABLE table=rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.559909 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae7895f7d7f47c3ab5fb0b93c1fd8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.560009 6867 tablet_service.cc:1467] Processing CreateTablet for tablet eef64b6b6db54db2b47314d672269043 (DEFAULT_TABLE table=rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.560318 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef64b6b6db54db2b47314d672269043. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.562191 8782 tablet_bootstrap.cc:492] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:05.562463 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 5365969434f142b8a5757a9a73bc109a (DEFAULT_TABLE table=rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.562733 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5365969434f142b8a5757a9a73bc109a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.562853 6999 tablet_service.cc:1467] Processing CreateTablet for tablet eef64b6b6db54db2b47314d672269043 (DEFAULT_TABLE table=rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.563082 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef64b6b6db54db2b47314d672269043. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.563516 8782 tablet_bootstrap.cc:654] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.564549 8782 tablet_bootstrap.cc:492] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:05.564627 8782 ts_tablet_manager.cc:1367] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:05.565078 8782 raft_consensus.cc:348] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.565209 8782 raft_consensus.cc:374] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.565243 8782 raft_consensus.cc:729] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.565428 8782 consensus_queue.cc:260] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.565827 8782 ts_tablet_manager.cc:1398] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.565840 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 5365969434f142b8a5757a9a73bc109a (DEFAULT_TABLE table=rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.566123 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5365969434f142b8a5757a9a73bc109a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.568514 8786 tablet_bootstrap.cc:492] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:05.568737 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 (DEFAULT_TABLE table=rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.568994 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae7895f7d7f47c3ab5fb0b93c1fd8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.569543 8786 tablet_bootstrap.cc:654] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.572086 7262 tablet_service.cc:1467] Processing CreateTablet for tablet eef64b6b6db54db2b47314d672269043 (DEFAULT_TABLE table=rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.572373 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef64b6b6db54db2b47314d672269043. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.574720 8782 tablet_bootstrap.cc:492] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:05.575794 8782 tablet_bootstrap.cc:654] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.576128 8784 tablet_bootstrap.cc:492] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:05.577391 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 (DEFAULT_TABLE table=rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.577858 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae7895f7d7f47c3ab5fb0b93c1fd8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.579180 8784 tablet_bootstrap.cc:654] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.580684 8783 tablet_bootstrap.cc:492] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:05.581693 8783 tablet_bootstrap.cc:654] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.583536 8786 tablet_bootstrap.cc:492] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:05.583698 8786 ts_tablet_manager.cc:1367] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:05.584072 8782 tablet_bootstrap.cc:492] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:05.584178 8782 ts_tablet_manager.cc:1367] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:05.584210 8786 raft_consensus.cc:348] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.584350 8786 raft_consensus.cc:374] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.584378 8786 raft_consensus.cc:729] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.584537 8786 consensus_queue.cc:260] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.584697 8782 raft_consensus.cc:348] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.584821 8782 raft_consensus.cc:374] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.584851 8782 raft_consensus.cc:729] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.584947 8786 ts_tablet_manager.cc:1398] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.584995 8782 consensus_queue.cc:260] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.585352 8782 ts_tablet_manager.cc:1398] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.585636 8786 tablet_bootstrap.cc:492] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:05.586799 8786 tablet_bootstrap.cc:654] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.587364 8783 tablet_bootstrap.cc:492] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:05.587393 8784 tablet_bootstrap.cc:492] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:05.587492 8784 ts_tablet_manager.cc:1367] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:05.587538 8783 ts_tablet_manager.cc:1367] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:05.588047 8784 raft_consensus.cc:348] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.588253 8784 raft_consensus.cc:374] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.588335 8784 raft_consensus.cc:729] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.588552 8784 consensus_queue.cc:260] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.589041 8784 ts_tablet_manager.cc:1398] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.588752 8783 raft_consensus.cc:348] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.589601 8783 raft_consensus.cc:374] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.589700 8783 raft_consensus.cc:729] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.590058 8783 consensus_queue.cc:260] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.592424 8783 ts_tablet_manager.cc:1398] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:05.593228 8783 tablet_bootstrap.cc:492] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:05.594511 8783 tablet_bootstrap.cc:654] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.596012 8782 tablet_bootstrap.cc:492] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:05.597584 8782 tablet_bootstrap.cc:654] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.597854 8783 tablet_bootstrap.cc:492] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:05.597862 8786 tablet_bootstrap.cc:492] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:05.597950 8783 ts_tablet_manager.cc:1367] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 07:59:05.597950 8786 ts_tablet_manager.cc:1367] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:05.598492 8786 raft_consensus.cc:348] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.598654 8786 raft_consensus.cc:374] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.598693 8786 raft_consensus.cc:729] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.598625 8783 raft_consensus.cc:348] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.598737 8783 raft_consensus.cc:374] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.598764 8783 raft_consensus.cc:729] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.598846 8786 consensus_queue.cc:260] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.598883 8783 consensus_queue.cc:260] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.599164 8786 ts_tablet_manager.cc:1398] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.599983 8783 ts_tablet_manager.cc:1398] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:05.600204 8783 tablet_bootstrap.cc:492] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:05.601476 8782 tablet_bootstrap.cc:492] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:05.601585 8782 ts_tablet_manager.cc:1367] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:05.601302 8783 tablet_bootstrap.cc:654] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.602098 8782 raft_consensus.cc:348] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.602231 8782 raft_consensus.cc:374] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.602267 8782 raft_consensus.cc:729] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.602454 8782 consensus_queue.cc:260] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.602761 8782 ts_tablet_manager.cc:1398] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.603586 8783 tablet_bootstrap.cc:492] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:05.603857 8783 ts_tablet_manager.cc:1367] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:05.604333 8783 raft_consensus.cc:348] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.604437 8783 raft_consensus.cc:374] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.604465 8783 raft_consensus.cc:729] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.604579 8783 consensus_queue.cc:260] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.604853 8783 ts_tablet_manager.cc:1398] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:05.623126 8456 raft_consensus.cc:482] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.623282 8456 raft_consensus.cc:504] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.623705 8456 leader_election.cc:288] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:05.624141 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5365969434f142b8a5757a9a73bc109a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:05.624142 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5365969434f142b8a5757a9a73bc109a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:05.624294 7020 raft_consensus.cc:2454] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:05.624297 7278 raft_consensus.cc:2454] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:05.624601 7086 leader_election.cc:302] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:05.624775 8456 raft_consensus.cc:2781] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.624828 8456 raft_consensus.cc:482] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.624853 8456 raft_consensus.cc:3037] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.625728 8456 raft_consensus.cc:504] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.626073 8456 leader_election.cc:288] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:05.626444 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5365969434f142b8a5757a9a73bc109a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:05.626477 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5365969434f142b8a5757a9a73bc109a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:05.626582 7020 raft_consensus.cc:3037] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.626605 7278 raft_consensus.cc:3037] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.627544 7020 raft_consensus.cc:2454] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:05.627607 7278 raft_consensus.cc:2454] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:05.627871 7086 leader_election.cc:302] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:05.628029 8456 raft_consensus.cc:2781] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.628146 8456 raft_consensus.cc:686] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:05.628304 8456 consensus_queue.cc:237] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.630474 6746 catalog_manager.cc:5484] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:05.671017 8765 raft_consensus.cc:482] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.671160 8765 raft_consensus.cc:504] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.671646 8765 leader_election.cc:288] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:05.672080 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae7895f7d7f47c3ab5fb0b93c1fd8e0" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:05.672083 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae7895f7d7f47c3ab5fb0b93c1fd8e0" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:05.672230 7149 raft_consensus.cc:2454] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:05.672238 7020 raft_consensus.cc:2454] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:05.672531 7217 leader_election.cc:302] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:05.672727 8765 raft_consensus.cc:2781] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.672786 8765 raft_consensus.cc:482] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.672811 8765 raft_consensus.cc:3037] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.673748 8765 raft_consensus.cc:504] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.674151 8765 leader_election.cc:288] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:05.674616 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae7895f7d7f47c3ab5fb0b93c1fd8e0" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:05.674624 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae7895f7d7f47c3ab5fb0b93c1fd8e0" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:05.674747 7020 raft_consensus.cc:3037] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.674752 7149 raft_consensus.cc:3037] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.675590 7149 raft_consensus.cc:2454] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:05.675596 7020 raft_consensus.cc:2454] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:05.675907 7218 leader_election.cc:302] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:05.676080 8765 raft_consensus.cc:2781] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.676226 8765 raft_consensus.cc:686] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:05.676551 8765 consensus_queue.cc:237] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:05.679039 6746 catalog_manager.cc:5484] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:05.812032 8192 raft_consensus.cc:482] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.812194 8192 raft_consensus.cc:504] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.812669 8192 leader_election.cc:288] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:05.813172 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef64b6b6db54db2b47314d672269043" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:05.813195 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef64b6b6db54db2b47314d672269043" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:05.813336 7279 raft_consensus.cc:2454] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:05.813364 7020 raft_consensus.cc:2454] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:05.813650 6825 leader_election.cc:302] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:05.813824 8192 raft_consensus.cc:2781] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.813879 8192 raft_consensus.cc:482] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.813905 8192 raft_consensus.cc:3037] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.814858 8192 raft_consensus.cc:504] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.815222 8192 leader_election.cc:288] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:05.815600 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef64b6b6db54db2b47314d672269043" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:05.815639 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef64b6b6db54db2b47314d672269043" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:05.815718 7020 raft_consensus.cc:3037] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.815745 7279 raft_consensus.cc:3037] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.816506 7020 raft_consensus.cc:2454] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:05.816516 7279 raft_consensus.cc:2454] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:05.816835 6825 leader_election.cc:302] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:05.817078 8192 raft_consensus.cc:2781] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.817184 8192 raft_consensus.cc:686] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:05.817327 8192 consensus_queue.cc:237] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:05.819332 6746 catalog_manager.cc:5484] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
W20240503 07:59:05.985937 8798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:05.986305 8798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:05.991088 8798 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240503 07:59:06.072659 8608 consensus_queue.cc:1035] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.076584 8456 consensus_queue.cc:1035] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.091446 7020 raft_consensus.cc:1261] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.092058 8765 consensus_queue.cc:1035] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.092348 7149 raft_consensus.cc:1261] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.092751 8765 consensus_queue.cc:1035] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.096894 8825 mvcc.cc:205] Tried to move back new op lower bound from 7023506006385475584 to 7023506004692664320. Current Snapshot: MvccSnapshot[applied={T|T < 7023506006385475584}]
I20240503 07:59:06.112545 7279 raft_consensus.cc:1261] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.112968 8589 consensus_queue.cc:1035] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.114001 7020 raft_consensus.cc:1261] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.114465 8192 consensus_queue.cc:1035] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 128.32 ms
time per row: 0.0641598 ms
I20240503 07:59:06.169301 6746 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:06.175801 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 30e7227748a940e88984e92e959b4446 (DEFAULT_TABLE table=rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.175801 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 30e7227748a940e88984e92e959b4446 (DEFAULT_TABLE table=rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.176139 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e7227748a940e88984e92e959b4446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.176141 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e7227748a940e88984e92e959b4446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.176560 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 1abaf2a559f94eb1a188b6263601e07a (DEFAULT_TABLE table=rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.176836 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abaf2a559f94eb1a188b6263601e07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.179083 8861 tablet_bootstrap.cc:492] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:06.179726 8862 tablet_bootstrap.cc:492] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:06.179803 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 30e7227748a940e88984e92e959b4446 (DEFAULT_TABLE table=rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.179867 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 7f946926ddd84a2cb758a795638e3ef5 (DEFAULT_TABLE table=rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.180096 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f946926ddd84a2cb758a795638e3ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.180105 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e7227748a940e88984e92e959b4446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.180748 8862 tablet_bootstrap.cc:654] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.182514 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 1abaf2a559f94eb1a188b6263601e07a (DEFAULT_TABLE table=rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.182653 8861 tablet_bootstrap.cc:654] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.182763 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abaf2a559f94eb1a188b6263601e07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.183660 8861 tablet_bootstrap.cc:492] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:06.183746 8861 ts_tablet_manager.cc:1367] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:06.184216 8861 raft_consensus.cc:348] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.184324 8861 raft_consensus.cc:374] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.184355 8861 raft_consensus.cc:729] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.184496 8861 consensus_queue.cc:260] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.184957 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 7f946926ddd84a2cb758a795638e3ef5 (DEFAULT_TABLE table=rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.185117 8861 ts_tablet_manager.cc:1398] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.185194 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f946926ddd84a2cb758a795638e3ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.185688 8861 tablet_bootstrap.cc:492] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:06.186902 8861 tablet_bootstrap.cc:654] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.187633 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 1abaf2a559f94eb1a188b6263601e07a (DEFAULT_TABLE table=rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.187897 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abaf2a559f94eb1a188b6263601e07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.187994 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 7f946926ddd84a2cb758a795638e3ef5 (DEFAULT_TABLE table=rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.188251 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f946926ddd84a2cb758a795638e3ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.192004 8860 tablet_bootstrap.cc:492] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:06.193017 8860 tablet_bootstrap.cc:654] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.193593 8864 tablet_bootstrap.cc:492] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:06.194706 8864 tablet_bootstrap.cc:654] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.195992 8864 tablet_bootstrap.cc:492] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:06.196094 8864 ts_tablet_manager.cc:1367] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:06.196552 8864 raft_consensus.cc:348] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.196674 8864 raft_consensus.cc:374] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.196712 8864 raft_consensus.cc:729] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.196857 8864 consensus_queue.cc:260] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.197153 8864 ts_tablet_manager.cc:1398] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:06.197384 8864 tablet_bootstrap.cc:492] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:06.198534 8864 tablet_bootstrap.cc:654] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.199770 8862 tablet_bootstrap.cc:492] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:06.199863 8864 tablet_bootstrap.cc:492] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:06.199879 8862 ts_tablet_manager.cc:1367] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 07:59:06.199949 8864 ts_tablet_manager.cc:1367] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:06.200368 8862 raft_consensus.cc:348] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.200480 8862 raft_consensus.cc:374] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.200428 8864 raft_consensus.cc:348] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.200507 8862 raft_consensus.cc:729] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.200541 8864 raft_consensus.cc:374] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.200572 8864 raft_consensus.cc:729] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.200634 8862 consensus_queue.cc:260] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.200717 8864 consensus_queue.cc:260] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.200909 8862 ts_tablet_manager.cc:1398] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.201242 8864 ts_tablet_manager.cc:1398] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.201429 8864 tablet_bootstrap.cc:492] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:06.202877 8864 tablet_bootstrap.cc:654] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.204123 8864 tablet_bootstrap.cc:492] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:06.204228 8864 ts_tablet_manager.cc:1367] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:06.204687 8864 raft_consensus.cc:348] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.204809 8864 raft_consensus.cc:374] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.204838 8864 raft_consensus.cc:729] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.204959 8864 consensus_queue.cc:260] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.205269 8864 ts_tablet_manager.cc:1398] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.205579 8861 tablet_bootstrap.cc:492] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:06.205695 8861 ts_tablet_manager.cc:1367] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 07:59:06.206175 8861 raft_consensus.cc:348] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.206363 8861 raft_consensus.cc:374] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.206426 8861 raft_consensus.cc:729] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.206614 8861 consensus_queue.cc:260] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.207127 8860 tablet_bootstrap.cc:492] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:06.207247 8860 ts_tablet_manager.cc:1367] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:06.207762 8860 raft_consensus.cc:348] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.207847 8861 ts_tablet_manager.cc:1398] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:06.207890 8860 raft_consensus.cc:374] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.207921 8860 raft_consensus.cc:729] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.208047 8860 consensus_queue.cc:260] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.208321 8860 ts_tablet_manager.cc:1398] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.208528 8860 tablet_bootstrap.cc:492] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:06.209563 8860 tablet_bootstrap.cc:654] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.211319 8860 tablet_bootstrap.cc:492] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:06.211416 8860 ts_tablet_manager.cc:1367] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:06.211906 8860 raft_consensus.cc:348] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.212016 8860 raft_consensus.cc:374] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.212056 8860 raft_consensus.cc:729] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.212215 8860 consensus_queue.cc:260] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.212559 8860 ts_tablet_manager.cc:1398] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.212774 8860 tablet_bootstrap.cc:492] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:06.213770 8860 tablet_bootstrap.cc:654] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.215122 8860 tablet_bootstrap.cc:492] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:06.215209 8860 ts_tablet_manager.cc:1367] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:06.215595 8860 raft_consensus.cc:348] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.215694 8860 raft_consensus.cc:374] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.215723 8860 raft_consensus.cc:729] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.215859 8860 consensus_queue.cc:260] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.216123 8860 ts_tablet_manager.cc:1398] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:06.244076 8192 raft_consensus.cc:482] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:06.244225 8192 raft_consensus.cc:504] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.244685 8192 leader_election.cc:288] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:06.245136 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f946926ddd84a2cb758a795638e3ef5" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:06.245157 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f946926ddd84a2cb758a795638e3ef5" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:06.245290 7020 raft_consensus.cc:2454] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:06.245293 7149 raft_consensus.cc:2454] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:06.245600 6824 leader_election.cc:302] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:06.245810 8192 raft_consensus.cc:2781] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:06.245887 8192 raft_consensus.cc:482] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:06.245934 8192 raft_consensus.cc:3037] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.246843 8192 raft_consensus.cc:504] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.247215 8192 leader_election.cc:288] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:06.247594 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f946926ddd84a2cb758a795638e3ef5" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:06.247594 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f946926ddd84a2cb758a795638e3ef5" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:06.247731 7149 raft_consensus.cc:3037] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.247732 7020 raft_consensus.cc:3037] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.248805 7149 raft_consensus.cc:2454] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:06.248805 7020 raft_consensus.cc:2454] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:06.249169 6824 leader_election.cc:302] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:06.249353 8589 raft_consensus.cc:2781] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:06.249433 8589 raft_consensus.cc:686] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:06.249574 8589 consensus_queue.cc:237] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.251605 6747 catalog_manager.cc:5484] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:06.321573 8826 raft_consensus.cc:482] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:06.321733 8826 raft_consensus.cc:504] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.322170 8826 leader_election.cc:288] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:06.322666 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e7227748a940e88984e92e959b4446" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:06.322662 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e7227748a940e88984e92e959b4446" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:06.322808 6889 raft_consensus.cc:2454] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:06.322821 7149 raft_consensus.cc:2454] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:06.323110 6956 leader_election.cc:302] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:06.323294 8826 raft_consensus.cc:2781] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:06.323346 8826 raft_consensus.cc:482] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:06.323371 8826 raft_consensus.cc:3037] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.324260 8826 raft_consensus.cc:504] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.324597 8826 leader_election.cc:288] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:06.324975 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e7227748a940e88984e92e959b4446" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:06.324999 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e7227748a940e88984e92e959b4446" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:06.325088 7149 raft_consensus.cc:3037] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.325099 6889 raft_consensus.cc:3037] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.325886 6889 raft_consensus.cc:2454] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:06.325901 7149 raft_consensus.cc:2454] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:06.326192 6956 leader_election.cc:302] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:06.326385 8826 raft_consensus.cc:2781] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:06.326478 8826 raft_consensus.cc:686] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:06.326615 8826 consensus_queue.cc:237] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.328763 6747 catalog_manager.cc:5484] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:06.335654 8855 raft_consensus.cc:482] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:06.335773 8855 raft_consensus.cc:504] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.336213 8855 leader_election.cc:288] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:06.336593 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abaf2a559f94eb1a188b6263601e07a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:06.336622 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abaf2a559f94eb1a188b6263601e07a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:06.336731 6889 raft_consensus.cc:2454] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:06.336758 7279 raft_consensus.cc:2454] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:06.337029 7084 leader_election.cc:302] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:06.337213 8855 raft_consensus.cc:2781] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:06.337275 8855 raft_consensus.cc:482] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:06.337312 8855 raft_consensus.cc:3037] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.338217 8855 raft_consensus.cc:504] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.338582 8855 leader_election.cc:288] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:06.338953 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abaf2a559f94eb1a188b6263601e07a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:06.338972 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abaf2a559f94eb1a188b6263601e07a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:06.339085 7279 raft_consensus.cc:3037] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.339085 6889 raft_consensus.cc:3037] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.340114 7279 raft_consensus.cc:2454] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:06.340190 6889 raft_consensus.cc:2454] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:06.340487 7087 leader_election.cc:302] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:06.340673 8855 raft_consensus.cc:2781] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:06.340791 8855 raft_consensus.cc:686] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:06.340937 8855 consensus_queue.cc:237] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.342986 6747 catalog_manager.cc:5484] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
W20240503 07:59:06.503325 8876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:06.503638 8876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:06.507303 8876 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240503 07:59:06.609439 7149 raft_consensus.cc:1261] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.610396 6889 raft_consensus.cc:1261] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.610595 8192 consensus_queue.cc:1035] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.610769 7149 raft_consensus.cc:1261] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.610846 7020 raft_consensus.cc:1261] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.611410 8826 consensus_queue.cc:1035] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.611711 8192 consensus_queue.cc:1035] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.612161 7279 raft_consensus.cc:1261] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.613271 6889 raft_consensus.cc:1261] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.613876 8855 consensus_queue.cc:1035] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.614249 8855 consensus_queue.cc:1035] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.617146 8826 consensus_queue.cc:1035] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.622152 8839 mvcc.cc:205] Tried to move back new op lower bound from 7023506008507944960 to 7023506007039713280. Current Snapshot: MvccSnapshot[applied={T|T < 7023506008507944960}]
INSERT report
rows total: 2000
time total: 131.528 ms
time per row: 0.065764 ms
I20240503 07:59:06.689816 6745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:06.696403 7260 tablet_service.cc:1467] Processing CreateTablet for tablet cb2190946fb04a6392097f9588ffda6b (DEFAULT_TABLE table=rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.696635 6868 tablet_service.cc:1467] Processing CreateTablet for tablet cb2190946fb04a6392097f9588ffda6b (DEFAULT_TABLE table=rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.696825 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2190946fb04a6392097f9588ffda6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.696946 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2190946fb04a6392097f9588ffda6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.699172 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 795f69249afe444ba0828875e1839b8c (DEFAULT_TABLE table=rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.699169 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 795f69249afe444ba0828875e1839b8c (DEFAULT_TABLE table=rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.699461 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795f69249afe444ba0828875e1839b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.699470 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795f69249afe444ba0828875e1839b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.699460 6998 tablet_service.cc:1467] Processing CreateTablet for tablet cb2190946fb04a6392097f9588ffda6b (DEFAULT_TABLE table=rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.699611 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 795f69249afe444ba0828875e1839b8c (DEFAULT_TABLE table=rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.699725 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2190946fb04a6392097f9588ffda6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.699848 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795f69249afe444ba0828875e1839b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.701869 8862 tablet_bootstrap.cc:492] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:06.701889 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 16c06ab5a3e54ed2910afa50069810cc (DEFAULT_TABLE table=rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.701994 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 16c06ab5a3e54ed2910afa50069810cc (DEFAULT_TABLE table=rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.702057 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 16c06ab5a3e54ed2910afa50069810cc (DEFAULT_TABLE table=rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.702134 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c06ab5a3e54ed2910afa50069810cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.702216 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c06ab5a3e54ed2910afa50069810cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.702279 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c06ab5a3e54ed2910afa50069810cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.702533 8860 tablet_bootstrap.cc:492] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:06.702565 8864 tablet_bootstrap.cc:492] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:06.703410 8860 tablet_bootstrap.cc:654] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.703410 8864 tablet_bootstrap.cc:654] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.704226 8861 tablet_bootstrap.cc:492] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:06.704735 8862 tablet_bootstrap.cc:654] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.705099 8861 tablet_bootstrap.cc:654] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.705807 8862 tablet_bootstrap.cc:492] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:06.705901 8862 ts_tablet_manager.cc:1367] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:06.706094 8861 tablet_bootstrap.cc:492] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:06.706179 8861 ts_tablet_manager.cc:1367] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:06.706393 8862 raft_consensus.cc:348] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.706506 8862 raft_consensus.cc:374] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.706537 8862 raft_consensus.cc:729] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.706658 8862 consensus_queue.cc:260] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.706696 8861 raft_consensus.cc:348] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.706805 8861 raft_consensus.cc:374] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.706833 8861 raft_consensus.cc:729] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.706907 8860 tablet_bootstrap.cc:492] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:06.706957 8862 ts_tablet_manager.cc:1398] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.706931 8861 consensus_queue.cc:260] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.706993 8860 ts_tablet_manager.cc:1367] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:06.707193 8861 ts_tablet_manager.cc:1398] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.707358 8861 tablet_bootstrap.cc:492] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:06.707463 8860 raft_consensus.cc:348] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.707592 8860 raft_consensus.cc:374] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.707624 8860 raft_consensus.cc:729] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.707754 8860 consensus_queue.cc:260] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.707914 8862 tablet_bootstrap.cc:492] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:06.708039 8860 ts_tablet_manager.cc:1398] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.708330 8861 tablet_bootstrap.cc:654] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.708791 8862 tablet_bootstrap.cc:654] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.708881 8860 tablet_bootstrap.cc:492] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:06.709878 8860 tablet_bootstrap.cc:654] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.711598 8860 tablet_bootstrap.cc:492] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:06.711699 8860 ts_tablet_manager.cc:1367] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:06.712304 8860 raft_consensus.cc:348] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.712466 8860 raft_consensus.cc:374] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.712497 8860 raft_consensus.cc:729] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.712699 8860 consensus_queue.cc:260] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.713158 8860 ts_tablet_manager.cc:1398] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.723366 8864 tablet_bootstrap.cc:492] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:06.723508 8864 ts_tablet_manager.cc:1367] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240503 07:59:06.724041 8864 raft_consensus.cc:348] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.724171 8864 raft_consensus.cc:374] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.724212 8864 raft_consensus.cc:729] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.724350 8864 consensus_queue.cc:260] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.724643 8864 ts_tablet_manager.cc:1398] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:06.724846 8864 tablet_bootstrap.cc:492] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:06.725991 8864 tablet_bootstrap.cc:654] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.727582 8861 tablet_bootstrap.cc:492] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:06.727692 8861 ts_tablet_manager.cc:1367] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 07:59:06.728243 8861 raft_consensus.cc:348] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.728381 8861 raft_consensus.cc:374] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.728416 8861 raft_consensus.cc:729] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.728552 8861 consensus_queue.cc:260] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.728973 8861 ts_tablet_manager.cc:1398] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.729111 8864 tablet_bootstrap.cc:492] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:06.729199 8864 ts_tablet_manager.cc:1367] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:06.729700 8864 raft_consensus.cc:348] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.729820 8864 raft_consensus.cc:374] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.729846 8864 raft_consensus.cc:729] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.729974 8864 consensus_queue.cc:260] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.730255 8864 ts_tablet_manager.cc:1398] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:06.731525 8862 tablet_bootstrap.cc:492] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:06.731657 8862 ts_tablet_manager.cc:1367] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 07:59:06.732250 8862 raft_consensus.cc:348] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.732391 8862 raft_consensus.cc:374] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.732419 8862 raft_consensus.cc:729] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.732545 8862 consensus_queue.cc:260] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:06.732875 8862 ts_tablet_manager.cc:1398] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:06.733080 8862 tablet_bootstrap.cc:492] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:06.734306 8862 tablet_bootstrap.cc:654] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.735625 8862 tablet_bootstrap.cc:492] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:06.735719 8862 ts_tablet_manager.cc:1367] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:06.736271 8862 raft_consensus.cc:348] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.736395 8862 raft_consensus.cc:374] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.736425 8862 raft_consensus.cc:729] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.736554 8862 consensus_queue.cc:260] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.736855 8862 ts_tablet_manager.cc:1398] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:06.853413 8916 raft_consensus.cc:482] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:06.853552 8916 raft_consensus.cc:504] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.854001 8916 leader_election.cc:288] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:06.854439 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c06ab5a3e54ed2910afa50069810cc" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:06.854463 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c06ab5a3e54ed2910afa50069810cc" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:06.854604 7020 raft_consensus.cc:2454] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:06.854588 7279 raft_consensus.cc:2454] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:06.854981 7086 leader_election.cc:302] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:06.855168 8916 raft_consensus.cc:2781] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:06.855248 8916 raft_consensus.cc:482] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:06.855278 8916 raft_consensus.cc:3037] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.856266 8916 raft_consensus.cc:504] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.856662 8916 leader_election.cc:288] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:06.857053 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c06ab5a3e54ed2910afa50069810cc" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:06.857069 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c06ab5a3e54ed2910afa50069810cc" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:06.857195 7020 raft_consensus.cc:3037] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.857199 7279 raft_consensus.cc:3037] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.858021 7020 raft_consensus.cc:2454] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:06.858029 7279 raft_consensus.cc:2454] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:06.858384 7086 leader_election.cc:302] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:06.858592 8916 raft_consensus.cc:2781] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:06.858676 8916 raft_consensus.cc:686] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:06.858829 8916 consensus_queue.cc:237] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:06.860957 6745 catalog_manager.cc:5484] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:06.954957 8589 raft_consensus.cc:482] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:06.955116 8589 raft_consensus.cc:504] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.955720 8589 leader_election.cc:288] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:06.956279 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2190946fb04a6392097f9588ffda6b" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:06.956265 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2190946fb04a6392097f9588ffda6b" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:06.956444 7279 raft_consensus.cc:2454] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:06.956444 7020 raft_consensus.cc:2454] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:06.956791 6825 leader_election.cc:302] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:06.956992 8589 raft_consensus.cc:2781] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:06.957069 8589 raft_consensus.cc:482] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:06.957098 8589 raft_consensus.cc:3037] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.958029 8589 raft_consensus.cc:504] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.958498 8589 leader_election.cc:288] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:06.958925 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2190946fb04a6392097f9588ffda6b" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:06.958933 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2190946fb04a6392097f9588ffda6b" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:06.959057 7279 raft_consensus.cc:3037] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.959051 7020 raft_consensus.cc:3037] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.960013 7279 raft_consensus.cc:2454] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:06.960038 7020 raft_consensus.cc:2454] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:06.960307 6825 leader_election.cc:302] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:06.960464 8192 raft_consensus.cc:2781] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:06.960529 8192 raft_consensus.cc:686] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:06.960664 8192 consensus_queue.cc:237] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:06.962798 6745 catalog_manager.cc:5484] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:07.032990 8916 raft_consensus.cc:482] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:07.033129 8916 raft_consensus.cc:504] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.033517 8916 leader_election.cc:288] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:07.034013 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795f69249afe444ba0828875e1839b8c" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:07.033986 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795f69249afe444ba0828875e1839b8c" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:07.034195 7279 raft_consensus.cc:2454] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:07.034195 6889 raft_consensus.cc:2454] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:07.034544 7087 leader_election.cc:302] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:07.034765 8916 raft_consensus.cc:2781] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:07.034837 8916 raft_consensus.cc:482] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:07.034881 8916 raft_consensus.cc:3037] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.035905 8916 raft_consensus.cc:504] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.036267 8916 leader_election.cc:288] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:07.036676 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795f69249afe444ba0828875e1839b8c" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:07.036676 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795f69249afe444ba0828875e1839b8c" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:07.036826 7279 raft_consensus.cc:3037] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.036826 6889 raft_consensus.cc:3037] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.037881 6889 raft_consensus.cc:2454] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:07.037881 7279 raft_consensus.cc:2454] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:07.038229 7087 leader_election.cc:302] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:07.038450 8916 raft_consensus.cc:2781] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:07.038542 8916 raft_consensus.cc:686] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:07.038719 8916 consensus_queue.cc:237] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.041015 6745 catalog_manager.cc:5484] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
W20240503 07:59:07.205030 8934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:07.205360 8934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:07.208711 8934 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240503 07:59:07.299491 7279 raft_consensus.cc:1261] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.299491 7020 raft_consensus.cc:1261] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.299919 7278 raft_consensus.cc:1261] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.299921 6889 raft_consensus.cc:1261] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.300410 8589 consensus_queue.cc:1035] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.300374 8920 consensus_queue.cc:1035] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.301281 8589 consensus_queue.cc:1035] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.302929 8916 consensus_queue.cc:1035] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.304054 7282 raft_consensus.cc:1261] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.305027 7018 raft_consensus.cc:1261] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.305763 8920 consensus_queue.cc:1035] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.306102 8920 consensus_queue.cc:1035] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.456 ms
time per row: 0.053728 ms
I20240503 07:59:07.370087 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:07.376572 6867 tablet_service.cc:1467] Processing CreateTablet for tablet ebc1fb490c6b448ab86bc707ee8faddc (DEFAULT_TABLE table=rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.376943 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc1fb490c6b448ab86bc707ee8faddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.377436 6999 tablet_service.cc:1467] Processing CreateTablet for tablet ebc1fb490c6b448ab86bc707ee8faddc (DEFAULT_TABLE table=rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.377475 7261 tablet_service.cc:1467] Processing CreateTablet for tablet ebc1fb490c6b448ab86bc707ee8faddc (DEFAULT_TABLE table=rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.377492 7262 tablet_service.cc:1467] Processing CreateTablet for tablet a00e3d35dbe5490f86bccd96a97f92f7 (DEFAULT_TABLE table=rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.377740 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc1fb490c6b448ab86bc707ee8faddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.377754 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc1fb490c6b448ab86bc707ee8faddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.377879 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00e3d35dbe5490f86bccd96a97f92f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.378651 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 1564dc8b294e4c7da812cd4fa6d510da (DEFAULT_TABLE table=rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.378901 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1564dc8b294e4c7da812cd4fa6d510da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.380370 8999 tablet_bootstrap.cc:492] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:07.380417 6869 tablet_service.cc:1467] Processing CreateTablet for tablet a00e3d35dbe5490f86bccd96a97f92f7 (DEFAULT_TABLE table=rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.380587 8998 tablet_bootstrap.cc:492] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:07.380671 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00e3d35dbe5490f86bccd96a97f92f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.381341 8999 tablet_bootstrap.cc:654] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.381633 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 1564dc8b294e4c7da812cd4fa6d510da (DEFAULT_TABLE table=rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.381700 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 1564dc8b294e4c7da812cd4fa6d510da (DEFAULT_TABLE table=rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.381930 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1564dc8b294e4c7da812cd4fa6d510da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.381955 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1564dc8b294e4c7da812cd4fa6d510da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.382699 7129 tablet_service.cc:1467] Processing CreateTablet for tablet a00e3d35dbe5490f86bccd96a97f92f7 (DEFAULT_TABLE table=rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.382941 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00e3d35dbe5490f86bccd96a97f92f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.384524 9002 tablet_bootstrap.cc:492] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:07.384750 8998 tablet_bootstrap.cc:654] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.385540 9000 tablet_bootstrap.cc:492] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:07.385718 9002 tablet_bootstrap.cc:654] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.386667 9000 tablet_bootstrap.cc:654] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.386889 9002 tablet_bootstrap.cc:492] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:07.386981 9002 ts_tablet_manager.cc:1367] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:07.387462 9002 raft_consensus.cc:348] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:07.387569 9002 raft_consensus.cc:374] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.387598 9002 raft_consensus.cc:729] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.387740 9002 consensus_queue.cc:260] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:07.388034 9002 ts_tablet_manager.cc:1398] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.388211 9002 tablet_bootstrap.cc:492] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:07.388597 8999 tablet_bootstrap.cc:492] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:07.388695 8999 ts_tablet_manager.cc:1367] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:07.388962 8998 tablet_bootstrap.cc:492] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:07.389056 8998 ts_tablet_manager.cc:1367] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:07.389094 9002 tablet_bootstrap.cc:654] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.389161 8999 raft_consensus.cc:348] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:07.389264 8999 raft_consensus.cc:374] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.389290 8999 raft_consensus.cc:729] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.389401 8999 consensus_queue.cc:260] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:07.389534 8998 raft_consensus.cc:348] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.389645 8998 raft_consensus.cc:374] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.389658 8999 ts_tablet_manager.cc:1398] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.389674 8998 raft_consensus.cc:729] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.389817 8999 tablet_bootstrap.cc:492] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:07.389827 8998 consensus_queue.cc:260] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.390165 8998 ts_tablet_manager.cc:1398] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.390337 8998 tablet_bootstrap.cc:492] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:07.390738 8999 tablet_bootstrap.cc:654] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.390996 9002 tablet_bootstrap.cc:492] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:07.391075 9002 ts_tablet_manager.cc:1367] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:07.391667 9002 raft_consensus.cc:348] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:07.391763 9002 raft_consensus.cc:374] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.391773 8998 tablet_bootstrap.cc:654] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.391794 9002 raft_consensus.cc:729] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.391892 9002 consensus_queue.cc:260] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:07.392176 9002 ts_tablet_manager.cc:1398] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.393654 9000 tablet_bootstrap.cc:492] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:07.393795 9000 ts_tablet_manager.cc:1367] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:07.394564 8999 tablet_bootstrap.cc:492] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:07.394568 9000 raft_consensus.cc:348] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:07.394665 8999 ts_tablet_manager.cc:1367] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:07.394687 9000 raft_consensus.cc:374] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.394721 9000 raft_consensus.cc:729] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.394879 9000 consensus_queue.cc:260] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:07.395078 8999 raft_consensus.cc:348] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.395171 8999 raft_consensus.cc:374] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.395197 8999 raft_consensus.cc:729] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.395301 8999 consensus_queue.cc:260] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.395435 9000 ts_tablet_manager.cc:1398] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.395588 8999 ts_tablet_manager.cc:1398] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.396574 8998 tablet_bootstrap.cc:492] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:07.396661 8998 ts_tablet_manager.cc:1367] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:07.398000 8998 raft_consensus.cc:348] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:07.398130 8998 raft_consensus.cc:374] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.398165 8998 raft_consensus.cc:729] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.398290 8998 consensus_queue.cc:260] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:07.398633 8998 ts_tablet_manager.cc:1398] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:07.398844 9000 tablet_bootstrap.cc:492] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:07.399631 8998 tablet_bootstrap.cc:492] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:07.400708 8998 tablet_bootstrap.cc:654] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.401559 9000 tablet_bootstrap.cc:654] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.403095 9000 tablet_bootstrap.cc:492] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:07.403203 9000 ts_tablet_manager.cc:1367] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:07.403460 8998 tablet_bootstrap.cc:492] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:07.403550 8998 ts_tablet_manager.cc:1367] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:07.403708 9000 raft_consensus.cc:348] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.403836 9000 raft_consensus.cc:374] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.403942 9000 raft_consensus.cc:729] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.404035 8998 raft_consensus.cc:348] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:07.404165 8998 raft_consensus.cc:374] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.404213 8998 raft_consensus.cc:729] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.404184 9000 consensus_queue.cc:260] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.404353 8998 consensus_queue.cc:260] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:07.404696 8998 ts_tablet_manager.cc:1398] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.405380 9000 ts_tablet_manager.cc:1398] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:07.410157 8765 raft_consensus.cc:482] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:07.410305 8765 raft_consensus.cc:504] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.410806 8765 leader_election.cc:288] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:07.411167 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc1fb490c6b448ab86bc707ee8faddc" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:07.411149 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc1fb490c6b448ab86bc707ee8faddc" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:07.411316 7018 raft_consensus.cc:2454] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:07.411320 6889 raft_consensus.cc:2454] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:07.411652 7215 leader_election.cc:302] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:07.411979 8765 raft_consensus.cc:2781] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:07.412039 8765 raft_consensus.cc:482] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:07.412076 8765 raft_consensus.cc:3037] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.413288 8765 raft_consensus.cc:504] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.413691 8765 leader_election.cc:288] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:07.414047 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc1fb490c6b448ab86bc707ee8faddc" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:07.414197 6889 raft_consensus.cc:3037] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.414155 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc1fb490c6b448ab86bc707ee8faddc" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:07.414295 7018 raft_consensus.cc:3037] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.415181 6889 raft_consensus.cc:2454] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:07.415197 7018 raft_consensus.cc:2454] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:07.415525 7215 leader_election.cc:302] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:07.415916 8765 raft_consensus.cc:2781] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:07.416023 8765 raft_consensus.cc:686] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:07.416175 8765 consensus_queue.cc:237] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.418458 6747 catalog_manager.cc:5484] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:07.438288 8969 raft_consensus.cc:482] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:07.438503 8969 raft_consensus.cc:504] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:07.439050 8969 leader_election.cc:288] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:07.439505 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1564dc8b294e4c7da812cd4fa6d510da" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:07.439551 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1564dc8b294e4c7da812cd4fa6d510da" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:07.439672 6889 raft_consensus.cc:2454] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:07.439672 7018 raft_consensus.cc:2454] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:07.440006 7086 leader_election.cc:302] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:07.440227 8969 raft_consensus.cc:2781] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:07.440299 8969 raft_consensus.cc:482] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:07.440354 8969 raft_consensus.cc:3037] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.441370 8969 raft_consensus.cc:504] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:07.441763 8969 leader_election.cc:288] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:07.442150 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1564dc8b294e4c7da812cd4fa6d510da" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:07.442159 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1564dc8b294e4c7da812cd4fa6d510da" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:07.442274 7018 raft_consensus.cc:3037] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.442289 6889 raft_consensus.cc:3037] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.443286 7018 raft_consensus.cc:2454] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:07.443459 6889 raft_consensus.cc:2454] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:07.443578 7086 leader_election.cc:302] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:07.443755 8969 raft_consensus.cc:2781] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:07.443861 8969 raft_consensus.cc:686] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:07.444036 8969 consensus_queue.cc:237] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:07.446084 6747 catalog_manager.cc:5484] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:07.589524 8765 raft_consensus.cc:482] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:07.589681 8765 raft_consensus.cc:504] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:07.590201 8765 leader_election.cc:288] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:07.590730 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00e3d35dbe5490f86bccd96a97f92f7" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:07.590725 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00e3d35dbe5490f86bccd96a97f92f7" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:07.590880 7151 raft_consensus.cc:2454] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:07.590886 6889 raft_consensus.cc:2454] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:07.591226 7218 leader_election.cc:302] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:07.591513 8765 raft_consensus.cc:2781] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:07.591584 8765 raft_consensus.cc:482] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:07.591626 8765 raft_consensus.cc:3037] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.592731 8765 raft_consensus.cc:504] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:07.593109 8765 leader_election.cc:288] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:07.593490 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00e3d35dbe5490f86bccd96a97f92f7" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:07.593533 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00e3d35dbe5490f86bccd96a97f92f7" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:07.593609 7151 raft_consensus.cc:3037] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.593674 6889 raft_consensus.cc:3037] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.594539 7151 raft_consensus.cc:2454] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:07.594584 6889 raft_consensus.cc:2454] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:07.594996 7215 leader_election.cc:302] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:07.595223 8765 raft_consensus.cc:2781] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:07.595330 8765 raft_consensus.cc:686] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:07.595495 8765 consensus_queue.cc:237] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:07.597719 6747 catalog_manager.cc:5484] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
W20240503 07:59:07.754608 9015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:07.754904 9015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:07.758234 9015 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240503 07:59:07.859836 7018 raft_consensus.cc:1261] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.860594 8765 consensus_queue.cc:1035] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.860761 6889 raft_consensus.cc:1261] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.862941 7151 raft_consensus.cc:1261] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.863258 8957 consensus_queue.cc:1035] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.863498 8765 consensus_queue.cc:1035] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.864023 6889 raft_consensus.cc:1261] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.867151 7020 raft_consensus.cc:1261] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.868857 8920 consensus_queue.cc:1035] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.871536 6889 raft_consensus.cc:1261] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.872370 8969 consensus_queue.cc:1035] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.877460 9053 consensus_queue.cc:1035] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 133.585 ms
time per row: 0.0667927 ms
I20240503 07:59:07.940969 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:07.947082 6999 tablet_service.cc:1467] Processing CreateTablet for tablet ba1de17d15b74c4497e649aadc8569d2 (DEFAULT_TABLE table=rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.947299 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 64d33ff4164b49a7948aa5afee40bf3f (DEFAULT_TABLE table=rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.947461 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1de17d15b74c4497e649aadc8569d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.947723 7129 tablet_service.cc:1467] Processing CreateTablet for tablet ba1de17d15b74c4497e649aadc8569d2 (DEFAULT_TABLE table=rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.948022 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1de17d15b74c4497e649aadc8569d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.948078 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3eb284c0fc48b4912168c434844284 (DEFAULT_TABLE table=rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.948187 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 64d33ff4164b49a7948aa5afee40bf3f (DEFAULT_TABLE table=rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.948347 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3eb284c0fc48b4912168c434844284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.948455 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d33ff4164b49a7948aa5afee40bf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.950121 7262 tablet_service.cc:1467] Processing CreateTablet for tablet ba1de17d15b74c4497e649aadc8569d2 (DEFAULT_TABLE table=rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.950418 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1de17d15b74c4497e649aadc8569d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.950832 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d33ff4164b49a7948aa5afee40bf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.950801 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3eb284c0fc48b4912168c434844284 (DEFAULT_TABLE table=rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.951026 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3eb284c0fc48b4912168c434844284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.951433 9079 tablet_bootstrap.cc:492] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:07.952418 9079 tablet_bootstrap.cc:654] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.952730 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 64d33ff4164b49a7948aa5afee40bf3f (DEFAULT_TABLE table=rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.952991 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d33ff4164b49a7948aa5afee40bf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.953291 9078 tablet_bootstrap.cc:492] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:07.954398 9078 tablet_bootstrap.cc:654] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.955060 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3eb284c0fc48b4912168c434844284 (DEFAULT_TABLE table=rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.955312 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3eb284c0fc48b4912168c434844284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.955348 9078 tablet_bootstrap.cc:492] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:07.955428 9078 ts_tablet_manager.cc:1367] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:07.955906 9078 raft_consensus.cc:348] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.956028 9078 raft_consensus.cc:374] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.956068 9078 raft_consensus.cc:729] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.956180 9078 consensus_queue.cc:260] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.956542 9078 ts_tablet_manager.cc:1398] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:07.956697 9078 tablet_bootstrap.cc:492] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:07.956748 9080 tablet_bootstrap.cc:492] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:07.957549 9078 tablet_bootstrap.cc:654] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.957715 9080 tablet_bootstrap.cc:654] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.957991 9077 tablet_bootstrap.cc:492] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:07.958884 9077 tablet_bootstrap.cc:654] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.959023 9079 tablet_bootstrap.cc:492] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:07.959131 9079 ts_tablet_manager.cc:1367] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:07.959607 9079 raft_consensus.cc:348] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.959748 9079 raft_consensus.cc:374] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.959781 9079 raft_consensus.cc:729] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.959901 9079 consensus_queue.cc:260] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.960080 9077 tablet_bootstrap.cc:492] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:07.960152 9077 ts_tablet_manager.cc:1367] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:07.960177 9079 ts_tablet_manager.cc:1398] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.960378 9079 tablet_bootstrap.cc:492] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:07.960590 9077 raft_consensus.cc:348] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.960701 9077 raft_consensus.cc:374] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.960731 9077 raft_consensus.cc:729] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.960852 9077 consensus_queue.cc:260] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.961122 9077 ts_tablet_manager.cc:1398] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.961227 9079 tablet_bootstrap.cc:654] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.961301 9077 tablet_bootstrap.cc:492] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:07.962117 9077 tablet_bootstrap.cc:654] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.962674 9079 tablet_bootstrap.cc:492] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:07.962754 9079 ts_tablet_manager.cc:1367] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:07.963176 9079 raft_consensus.cc:348] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.963286 9079 raft_consensus.cc:374] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.963316 9079 raft_consensus.cc:729] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.963446 9079 consensus_queue.cc:260] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.963913 9079 ts_tablet_manager.cc:1398] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.964087 9079 tablet_bootstrap.cc:492] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:07.965090 9079 tablet_bootstrap.cc:654] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.965621 9077 tablet_bootstrap.cc:492] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:07.965786 9077 ts_tablet_manager.cc:1367] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:07.966177 9079 tablet_bootstrap.cc:492] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:07.966253 9079 ts_tablet_manager.cc:1367] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:07.966390 9077 raft_consensus.cc:348] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.966498 9077 raft_consensus.cc:374] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.966526 9077 raft_consensus.cc:729] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.966650 9077 consensus_queue.cc:260] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.966859 9079 raft_consensus.cc:348] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.966970 9079 raft_consensus.cc:374] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.967006 9079 raft_consensus.cc:729] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.967231 9077 ts_tablet_manager.cc:1398] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.967232 9079 consensus_queue.cc:260] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.967514 9077 tablet_bootstrap.cc:492] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:07.967597 9079 ts_tablet_manager.cc:1398] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:07.968516 9077 tablet_bootstrap.cc:654] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.969193 9080 tablet_bootstrap.cc:492] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:07.969306 9080 ts_tablet_manager.cc:1367] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:07.969597 9078 tablet_bootstrap.cc:492] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:07.969658 9077 tablet_bootstrap.cc:492] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:07.969692 9078 ts_tablet_manager.cc:1367] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.001s
I20240503 07:59:07.969748 9077 ts_tablet_manager.cc:1367] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:07.970127 9078 raft_consensus.cc:348] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.970244 9078 raft_consensus.cc:374] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.970198 9077 raft_consensus.cc:348] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.970278 9078 raft_consensus.cc:729] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.970309 9077 raft_consensus.cc:374] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.970376 9077 raft_consensus.cc:729] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.970456 9078 consensus_queue.cc:260] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:07.970487 9077 consensus_queue.cc:260] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.970552 9080 raft_consensus.cc:348] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.970657 9080 raft_consensus.cc:374] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.970759 9077 ts_tablet_manager.cc:1398] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.970767 9080 raft_consensus.cc:729] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.970927 9080 consensus_queue.cc:260] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:07.971686 9078 ts_tablet_manager.cc:1398] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:07.975847 9080 ts_tablet_manager.cc:1398] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:08.003064 8976 raft_consensus.cc:482] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.003194 8976 raft_consensus.cc:504] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:08.003613 8976 leader_election.cc:288] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:08.004029 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3eb284c0fc48b4912168c434844284" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:08.004053 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3eb284c0fc48b4912168c434844284" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:08.004177 6889 raft_consensus.cc:2454] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:08.004182 7278 raft_consensus.cc:2454] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:08.004469 6953 leader_election.cc:302] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:08.004703 8976 raft_consensus.cc:2781] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.004756 8976 raft_consensus.cc:482] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.004781 8976 raft_consensus.cc:3037] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.005759 8976 raft_consensus.cc:504] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:08.006129 8976 leader_election.cc:288] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:08.006516 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3eb284c0fc48b4912168c434844284" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:08.006542 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3eb284c0fc48b4912168c434844284" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:08.006672 7278 raft_consensus.cc:3037] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.006671 6889 raft_consensus.cc:3037] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.007637 7278 raft_consensus.cc:2454] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:08.007763 6889 raft_consensus.cc:2454] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:08.007979 6956 leader_election.cc:302] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:08.008265 8976 raft_consensus.cc:2781] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.008486 8976 raft_consensus.cc:686] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:08.008657 8976 consensus_queue.cc:237] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:08.010648 6742 catalog_manager.cc:5484] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:08.183213 9053 raft_consensus.cc:482] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.183394 9053 raft_consensus.cc:504] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.183876 9053 leader_election.cc:288] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:08.184265 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1de17d15b74c4497e649aadc8569d2" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:08.184361 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1de17d15b74c4497e649aadc8569d2" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:08.184423 7151 raft_consensus.cc:2454] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:08.184517 7018 raft_consensus.cc:2454] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:08.184764 7218 leader_election.cc:302] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:08.184962 9053 raft_consensus.cc:2781] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.185026 9053 raft_consensus.cc:482] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.185063 9053 raft_consensus.cc:3037] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.186223 9053 raft_consensus.cc:504] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.186654 9053 leader_election.cc:288] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:08.186992 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1de17d15b74c4497e649aadc8569d2" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:08.186992 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1de17d15b74c4497e649aadc8569d2" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:08.187126 7018 raft_consensus.cc:3037] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.187158 7151 raft_consensus.cc:3037] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.188133 7018 raft_consensus.cc:2454] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:08.188141 7151 raft_consensus.cc:2454] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:08.188457 7218 leader_election.cc:302] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:08.188720 9053 raft_consensus.cc:2781] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.188899 9053 raft_consensus.cc:686] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:08.189035 9053 consensus_queue.cc:237] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.191157 6742 catalog_manager.cc:5484] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:08.211467 9053 raft_consensus.cc:482] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.211629 9053 raft_consensus.cc:504] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.212071 9053 leader_election.cc:288] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:08.212582 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d33ff4164b49a7948aa5afee40bf3f" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:08.212591 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d33ff4164b49a7948aa5afee40bf3f" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:08.212724 7018 raft_consensus.cc:2454] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:08.212739 7151 raft_consensus.cc:2454] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:08.213102 7218 leader_election.cc:302] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:08.213311 9053 raft_consensus.cc:2781] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.213393 9053 raft_consensus.cc:482] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.213419 9053 raft_consensus.cc:3037] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.214294 9053 raft_consensus.cc:504] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.214694 9053 leader_election.cc:288] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:08.215070 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d33ff4164b49a7948aa5afee40bf3f" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:08.215083 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d33ff4164b49a7948aa5afee40bf3f" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:08.215198 7018 raft_consensus.cc:3037] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.215198 7151 raft_consensus.cc:3037] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.216151 7151 raft_consensus.cc:2454] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:08.216166 7018 raft_consensus.cc:2454] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:08.216464 7217 leader_election.cc:302] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:08.216719 9053 raft_consensus.cc:2781] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.216852 9053 raft_consensus.cc:686] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:08.217196 9053 consensus_queue.cc:237] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.219357 6742 catalog_manager.cc:5484] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
W20240503 07:59:08.377265 9094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:08.377604 9094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:08.381913 9094 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240503 07:59:08.470436 8976 consensus_queue.cc:1035] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.472404 7151 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.472965 9053 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.473842 7020 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.473902 7018 raft_consensus.cc:1261] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.476087 9059 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.476437 9059 consensus_queue.cc:1035] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.477635 8976 consensus_queue.cc:1035] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.478477 7149 raft_consensus.cc:1261] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.479274 9059 consensus_queue.cc:1035] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 117.708 ms
time per row: 0.0588538 ms
I20240503 07:59:08.547382 6745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:08.553498 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 508adb2132b0497195c782a364bb1b40 (DEFAULT_TABLE table=rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.553743 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 508adb2132b0497195c782a364bb1b40 (DEFAULT_TABLE table=rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.553880 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508adb2132b0497195c782a364bb1b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.554050 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508adb2132b0497195c782a364bb1b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.554024 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 508adb2132b0497195c782a364bb1b40 (DEFAULT_TABLE table=rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.554323 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508adb2132b0497195c782a364bb1b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.556257 9154 tablet_bootstrap.cc:492] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:08.556413 6869 tablet_service.cc:1467] Processing CreateTablet for tablet b74fd6f87ab44bebb22167d0b6372cd1 (DEFAULT_TABLE table=rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.556648 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74fd6f87ab44bebb22167d0b6372cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.556756 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7b63d090d34715b52e64df9701d60d (DEFAULT_TABLE table=rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.557003 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7b63d090d34715b52e64df9701d60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.558630 7129 tablet_service.cc:1467] Processing CreateTablet for tablet b74fd6f87ab44bebb22167d0b6372cd1 (DEFAULT_TABLE table=rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.558889 9154 tablet_bootstrap.cc:654] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.558895 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74fd6f87ab44bebb22167d0b6372cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.559177 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7b63d090d34715b52e64df9701d60d (DEFAULT_TABLE table=rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.559372 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7b63d090d34715b52e64df9701d60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.560166 9154 tablet_bootstrap.cc:492] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:08.560245 9154 ts_tablet_manager.cc:1367] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:08.560690 9154 raft_consensus.cc:348] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:08.560814 9154 raft_consensus.cc:374] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.560845 9154 raft_consensus.cc:729] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.561200 9154 consensus_queue.cc:260] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:08.561177 7260 tablet_service.cc:1467] Processing CreateTablet for tablet b74fd6f87ab44bebb22167d0b6372cd1 (DEFAULT_TABLE table=rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.561480 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74fd6f87ab44bebb22167d0b6372cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.561820 9154 ts_tablet_manager.cc:1398] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.562913 9155 tablet_bootstrap.cc:492] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:08.563913 9158 tablet_bootstrap.cc:492] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:08.564114 9155 tablet_bootstrap.cc:654] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.564826 9158 tablet_bootstrap.cc:654] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.565290 9155 tablet_bootstrap.cc:492] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:08.565385 9155 ts_tablet_manager.cc:1367] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:08.565482 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7b63d090d34715b52e64df9701d60d (DEFAULT_TABLE table=rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.565733 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7b63d090d34715b52e64df9701d60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.565865 9155 raft_consensus.cc:348] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:08.565984 9155 raft_consensus.cc:374] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.566020 9158 tablet_bootstrap.cc:492] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:08.566045 9155 raft_consensus.cc:729] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.566102 9158 ts_tablet_manager.cc:1367] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:08.566170 9155 consensus_queue.cc:260] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:08.566644 9155 ts_tablet_manager.cc:1398] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.566593 9158 raft_consensus.cc:348] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.566700 9158 raft_consensus.cc:374] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.566730 9158 raft_consensus.cc:729] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.566843 9155 tablet_bootstrap.cc:492] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:08.566845 9158 consensus_queue.cc:260] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.567111 9158 ts_tablet_manager.cc:1398] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.567740 9155 tablet_bootstrap.cc:654] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.568854 9154 tablet_bootstrap.cc:492] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:08.571071 9154 tablet_bootstrap.cc:654] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.572137 9156 tablet_bootstrap.cc:492] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:08.572909 9154 tablet_bootstrap.cc:492] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:08.573042 9154 ts_tablet_manager.cc:1367] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:08.573400 9156 tablet_bootstrap.cc:654] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.573896 9154 raft_consensus.cc:348] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.574030 9154 raft_consensus.cc:374] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.574083 9154 raft_consensus.cc:729] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.574241 9154 consensus_queue.cc:260] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.574559 9154 ts_tablet_manager.cc:1398] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.574818 9154 tablet_bootstrap.cc:492] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:08.575120 9156 tablet_bootstrap.cc:492] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:08.575235 9156 ts_tablet_manager.cc:1367] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:08.575789 9154 tablet_bootstrap.cc:654] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.575752 9156 raft_consensus.cc:348] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:08.575907 9156 raft_consensus.cc:374] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.575942 9156 raft_consensus.cc:729] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.576087 9156 consensus_queue.cc:260] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:08.576385 9156 ts_tablet_manager.cc:1398] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:08.577361 9156 tablet_bootstrap.cc:492] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:08.578348 9156 tablet_bootstrap.cc:654] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.579552 9156 tablet_bootstrap.cc:492] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:08.579697 9156 ts_tablet_manager.cc:1367] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:08.580482 9156 raft_consensus.cc:348] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.580600 9156 raft_consensus.cc:374] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.580627 9156 raft_consensus.cc:729] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.580853 9156 consensus_queue.cc:260] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.581251 9156 ts_tablet_manager.cc:1398] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.581436 9156 tablet_bootstrap.cc:492] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:08.582466 9156 tablet_bootstrap.cc:654] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.583395 9155 tablet_bootstrap.cc:492] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:08.583494 9155 ts_tablet_manager.cc:1367] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 07:59:08.583868 9154 tablet_bootstrap.cc:492] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:08.583948 9154 ts_tablet_manager.cc:1367] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:08.584014 9155 raft_consensus.cc:348] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:08.584131 9155 raft_consensus.cc:374] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.584165 9155 raft_consensus.cc:729] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.584302 9155 consensus_queue.cc:260] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:08.584606 9155 ts_tablet_manager.cc:1398] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.584808 9154 raft_consensus.cc:348] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:08.584921 9154 raft_consensus.cc:374] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.584949 9154 raft_consensus.cc:729] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.585084 9154 consensus_queue.cc:260] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:08.585651 9154 ts_tablet_manager.cc:1398] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.587253 9156 tablet_bootstrap.cc:492] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:08.587417 9156 ts_tablet_manager.cc:1367] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:08.588351 9156 raft_consensus.cc:348] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:08.588483 9156 raft_consensus.cc:374] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.588536 9156 raft_consensus.cc:729] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.588874 9156 consensus_queue.cc:260] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:08.589254 9156 ts_tablet_manager.cc:1398] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:08.619527 8589 raft_consensus.cc:482] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.619685 8589 raft_consensus.cc:504] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.620126 8589 leader_election.cc:288] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:08.620618 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74fd6f87ab44bebb22167d0b6372cd1" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:08.620679 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74fd6f87ab44bebb22167d0b6372cd1" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:08.620775 7282 raft_consensus.cc:2454] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:08.620869 7149 raft_consensus.cc:2454] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:08.621129 6825 leader_election.cc:302] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:08.621327 8589 raft_consensus.cc:2781] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.621385 8589 raft_consensus.cc:482] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.621413 8589 raft_consensus.cc:3037] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.622432 8589 raft_consensus.cc:504] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.622817 8589 leader_election.cc:288] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:08.623389 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74fd6f87ab44bebb22167d0b6372cd1" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:08.623387 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74fd6f87ab44bebb22167d0b6372cd1" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:08.623520 7282 raft_consensus.cc:3037] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.623541 7149 raft_consensus.cc:3037] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.624565 7282 raft_consensus.cc:2454] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:08.624565 7149 raft_consensus.cc:2454] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:08.624869 6825 leader_election.cc:302] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:08.625047 8589 raft_consensus.cc:2781] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.625154 8589 raft_consensus.cc:686] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:08.625305 8589 consensus_queue.cc:237] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:08.627406 6743 catalog_manager.cc:5484] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:08.673945 9150 raft_consensus.cc:482] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.674090 9150 raft_consensus.cc:504] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:08.674654 9150 leader_election.cc:288] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:08.675113 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508adb2132b0497195c782a364bb1b40" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:08.675113 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508adb2132b0497195c782a364bb1b40" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:08.675266 6889 raft_consensus.cc:2454] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:08.675267 7149 raft_consensus.cc:2454] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:08.675609 6953 leader_election.cc:302] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:08.675791 9150 raft_consensus.cc:2781] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.675848 9150 raft_consensus.cc:482] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.675885 9150 raft_consensus.cc:3037] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.676889 9150 raft_consensus.cc:504] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:08.677280 9150 leader_election.cc:288] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:08.677649 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508adb2132b0497195c782a364bb1b40" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:08.677677 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508adb2132b0497195c782a364bb1b40" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:08.677799 7149 raft_consensus.cc:3037] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.677810 6889 raft_consensus.cc:3037] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.678615 6889 raft_consensus.cc:2454] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:08.678622 7149 raft_consensus.cc:2454] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:08.678947 6956 leader_election.cc:302] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:08.679119 9150 raft_consensus.cc:2781] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.679208 9150 raft_consensus.cc:686] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:08.679378 9150 consensus_queue.cc:237] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:08.681524 6743 catalog_manager.cc:5484] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:08.727900 9150 raft_consensus.cc:482] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.728044 9150 raft_consensus.cc:504] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:08.728474 9150 leader_election.cc:288] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:08.728933 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7b63d090d34715b52e64df9701d60d" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:08.728935 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7b63d090d34715b52e64df9701d60d" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:08.729075 6889 raft_consensus.cc:2454] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:08.729089 7149 raft_consensus.cc:2454] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:08.729405 6953 leader_election.cc:302] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:08.729570 9150 raft_consensus.cc:2781] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.729631 9150 raft_consensus.cc:482] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.729658 9150 raft_consensus.cc:3037] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.730629 9150 raft_consensus.cc:504] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:08.731024 9150 leader_election.cc:288] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:08.731411 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7b63d090d34715b52e64df9701d60d" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:08.731417 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7b63d090d34715b52e64df9701d60d" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:08.731552 6889 raft_consensus.cc:3037] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.731549 7149 raft_consensus.cc:3037] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.732462 7149 raft_consensus.cc:2454] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:08.732467 6889 raft_consensus.cc:2454] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:08.732822 6953 leader_election.cc:302] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:08.733001 9150 raft_consensus.cc:2781] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.733104 9150 raft_consensus.cc:686] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:08.733305 9150 consensus_queue.cc:237] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:08.735589 6743 catalog_manager.cc:5484] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
W20240503 07:59:08.889844 9170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:08.890161 9170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:08.893864 9170 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240503 07:59:08.998876 6889 raft_consensus.cc:1261] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.999678 7149 raft_consensus.cc:1261] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.999769 9150 consensus_queue.cc:1035] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.000182 9151 consensus_queue.cc:1035] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.002928 6888 raft_consensus.cc:1261] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.003686 9151 consensus_queue.cc:1035] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.004302 7282 raft_consensus.cc:1261] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.004515 9131 mvcc.cc:205] Tried to move back new op lower bound from 7023506018293215232 to 7023506016992333824. Current Snapshot: MvccSnapshot[applied={T|T < 7023506018293215232}]
I20240503 07:59:09.004746 8192 consensus_queue.cc:1035] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.005475 7151 raft_consensus.cc:1261] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.005777 7150 raft_consensus.cc:1261] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.011118 9151 consensus_queue.cc:1035] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.012228 9129 mvcc.cc:205] Tried to move back new op lower bound from 7023506018293215232 to 7023506016992333824. Current Snapshot: MvccSnapshot[applied={T|T < 7023506018293215232}]
I20240503 07:59:09.012333 8192 consensus_queue.cc:1035] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.022616 9123 mvcc.cc:205] Tried to move back new op lower bound from 7023506018310344704 to 7023506016770625536. Current Snapshot: MvccSnapshot[applied={T|T < 7023506018310344704}]
INSERT report
rows total: 2000
time total: 136.319 ms
time per row: 0.0681593 ms
I20240503 07:59:09.079483 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:09.086081 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 7bbaf13cbb474734a961664ae8ebe779 (DEFAULT_TABLE table=rebalance_test_table_19 [id=071fb84151514ba094acc7ff3c1b24e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:09.086149 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 7bbaf13cbb474734a961664ae8ebe779 (DEFAULT_TABLE table=rebalance_test_table_19 [id=071fb84151514ba094acc7ff3c1b24e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:09.086485 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbaf13cbb474734a961664ae8ebe779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.086513 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbaf13cbb474734a961664ae8ebe779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.086961 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 721c34a8b6a348e5890d462ba7da938c (DEFAULT_TABLE table=rebalance_test_table_19 [id=071fb84151514ba094acc7ff3c1b24e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:09.087497 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 7bbaf13cbb474734a961664ae8ebe779 (DEFAULT_TABLE table=rebalance_test_table_19 [id=071fb84151514ba094acc7ff3c1b24e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:09.087779 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbaf13cbb474734a961664ae8ebe779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.087783 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721c34a8b6a348e5890d462ba7da938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.088869 9214 tablet_bootstrap.cc:492] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:09.089423 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 721c34a8b6a348e5890d462ba7da938c (DEFAULT_TABLE table=rebalance_test_table_19 [id=071fb84151514ba094acc7ff3c1b24e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:09.089685 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721c34a8b6a348e5890d462ba7da938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.089852 9214 tablet_bootstrap.cc:654] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:09.090235 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 82fc9f4263244c6fa2fb39543230eb87 (DEFAULT_TABLE table=rebalance_test_table_19 [id=071fb84151514ba094acc7ff3c1b24e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:09.090235 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 721c34a8b6a348e5890d462ba7da938c (DEFAULT_TABLE table=rebalance_test_table_19 [id=071fb84151514ba094acc7ff3c1b24e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:09.090494 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fc9f4263244c6fa2fb39543230eb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.090494 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721c34a8b6a348e5890d462ba7da938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.091913 9217 tablet_bootstrap.cc:492] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:09.091991 9156 tablet_bootstrap.cc:492] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:09.092674 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 82fc9f4263244c6fa2fb39543230eb87 (DEFAULT_TABLE table=rebalance_test_table_19 [id=071fb84151514ba094acc7ff3c1b24e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:09.092924 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fc9f4263244c6fa2fb39543230eb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.092943 9156 tablet_bootstrap.cc:654] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:09.093657 9217 tablet_bootstrap.cc:654] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:09.094642 9156 tablet_bootstrap.cc:492] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:09.094744 9217 tablet_bootstrap.cc:492] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:09.094756 9156 ts_tablet_manager.cc:1367] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:09.094835 9217 ts_tablet_manager.cc:1367] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:09.095258 9217 raft_consensus.cc:348] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.095386 9217 raft_consensus.cc:374] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:09.095420 9217 raft_consensus.cc:729] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:09.095366 9156 raft_consensus.cc:348] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.095477 9156 raft_consensus.cc:374] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:09.095510 9156 raft_consensus.cc:729] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:09.095546 9217 consensus_queue.cc:260] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.095624 9156 consensus_queue.cc:260] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.095836 9217 ts_tablet_manager.cc:1398] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:09.095927 9156 ts_tablet_manager.cc:1398] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:09.095847 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 82fc9f4263244c6fa2fb39543230eb87 (DEFAULT_TABLE table=rebalance_test_table_19 [id=071fb84151514ba094acc7ff3c1b24e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:09.096041 9217 tablet_bootstrap.cc:492] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:09.096107 9156 tablet_bootstrap.cc:492] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:09.096150 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fc9f4263244c6fa2fb39543230eb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.097012 9217 tablet_bootstrap.cc:654] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:09.097047 9156 tablet_bootstrap.cc:654] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:09.097919 9214 tablet_bootstrap.cc:492] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:09.098011 9214 ts_tablet_manager.cc:1367] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:09.098192 9215 tablet_bootstrap.cc:492] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:09.098245 9156 tablet_bootstrap.cc:492] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:09.098312 9156 ts_tablet_manager.cc:1367] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:09.098511 9214 raft_consensus.cc:348] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.098618 9214 raft_consensus.cc:374] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:09.098649 9214 raft_consensus.cc:729] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:09.098754 9214 consensus_queue.cc:260] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.098755 9156 raft_consensus.cc:348] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:09.098886 9156 raft_consensus.cc:374] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:09.098919 9156 raft_consensus.cc:729] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:09.099033 9214 ts_tablet_manager.cc:1398] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:09.099032 9156 consensus_queue.cc:260] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:09.099202 9214 tablet_bootstrap.cc:492] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:09.099220 9215 tablet_bootstrap.cc:654] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:09.099332 9156 ts_tablet_manager.cc:1398] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:09.100127 9214 tablet_bootstrap.cc:654] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:09.100576 9215 tablet_bootstrap.cc:492] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:09.100687 9215 ts_tablet_manager.cc:1367] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:09.101137 9215 raft_consensus.cc:348] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.101248 9215 raft_consensus.cc:374] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:09.101279 9215 raft_consensus.cc:729] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:09.101406 9215 consensus_queue.cc:260] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.101692 9215 ts_tablet_manager.cc:1398] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:09.101881 9215 tablet_bootstrap.cc:492] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:09.102921 9215 tablet_bootstrap.cc:654] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:09.105048 9214 tablet_bootstrap.cc:492] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:09.105156 9214 ts_tablet_manager.cc:1367] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:09.105638 9214 raft_consensus.cc:348] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:09.105741 9214 raft_consensus.cc:374] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:09.105759 9214 raft_consensus.cc:729] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:09.105809 9215 tablet_bootstrap.cc:492] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:09.105857 9214 consensus_queue.cc:260] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:09.105891 9215 ts_tablet_manager.cc:1367] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:09.106105 9214 ts_tablet_manager.cc:1398] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:09.106242 9215 raft_consensus.cc:348] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.106325 9215 raft_consensus.cc:374] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:09.106384 9215 raft_consensus.cc:729] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:09.106509 9215 consensus_queue.cc:260] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.106761 9215 ts_tablet_manager.cc:1398] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:09.108225 9217 tablet_bootstrap.cc:492] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:09.108333 9217 ts_tablet_manager.cc:1367] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:09.108809 9217 raft_consensus.cc:348] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:09.108944 9217 raft_consensus.cc:374] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:09.108976 9217 raft_consensus.cc:729] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:09.109093 9217 consensus_queue.cc:260] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:09.109390 9217 ts_tablet_manager.cc:1398] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:09.109581 9217 tablet_bootstrap.cc:492] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:09.112511 9217 tablet_bootstrap.cc:654] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:09.113710 9217 tablet_bootstrap.cc:492] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:09.113799 9217 ts_tablet_manager.cc:1367] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:09.114286 9217 raft_consensus.cc:348] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.114435 9217 raft_consensus.cc:374] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:09.114464 9217 raft_consensus.cc:729] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:09.114590 9217 consensus_queue.cc:260] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.114876 9217 ts_tablet_manager.cc:1398] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:09.235138 9053 raft_consensus.cc:482] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:09.235299 9053 raft_consensus.cc:504] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.235889 9053 leader_election.cc:288] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:09.236307 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721c34a8b6a348e5890d462ba7da938c" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:09.236306 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721c34a8b6a348e5890d462ba7da938c" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:09.236466 6889 raft_consensus.cc:2454] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:09.236466 7150 raft_consensus.cc:2454] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:09.236770 7218 leader_election.cc:302] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:09.237002 9053 raft_consensus.cc:2781] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:09.237073 9053 raft_consensus.cc:482] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:09.237115 9053 raft_consensus.cc:3037] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:09.238081 9053 raft_consensus.cc:504] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.238577 9053 leader_election.cc:288] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:09.238834 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721c34a8b6a348e5890d462ba7da938c" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:09.238906 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721c34a8b6a348e5890d462ba7da938c" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:09.238960 6889 raft_consensus.cc:3037] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:09.239044 7150 raft_consensus.cc:3037] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:09.239903 6889 raft_consensus.cc:2454] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:09.239940 7150 raft_consensus.cc:2454] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:09.240271 7215 leader_election.cc:302] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:09.240474 9053 raft_consensus.cc:2781] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:09.240648 9053 raft_consensus.cc:686] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:09.240849 9053 consensus_queue.cc:237] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.243450 6744 catalog_manager.cc:5484] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:09.254072 9151 raft_consensus.cc:482] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:09.254258 9151 raft_consensus.cc:504] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:09.254806 9151 leader_election.cc:288] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:09.255308 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fc9f4263244c6fa2fb39543230eb87" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:09.255300 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fc9f4263244c6fa2fb39543230eb87" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:09.255467 7150 raft_consensus.cc:2454] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:09.255467 7278 raft_consensus.cc:2454] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:09.255757 6956 leader_election.cc:302] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:09.255972 9151 raft_consensus.cc:2781] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:09.256047 9151 raft_consensus.cc:482] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:09.256084 9151 raft_consensus.cc:3037] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:09.257019 9151 raft_consensus.cc:504] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:09.257387 9151 leader_election.cc:288] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:09.257800 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fc9f4263244c6fa2fb39543230eb87" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:09.257869 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fc9f4263244c6fa2fb39543230eb87" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:09.257947 7278 raft_consensus.cc:3037] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:09.257980 7150 raft_consensus.cc:3037] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:09.259042 7278 raft_consensus.cc:2454] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:09.259042 7150 raft_consensus.cc:2454] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:09.259405 6956 leader_election.cc:302] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:09.259642 9151 raft_consensus.cc:2781] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:09.259738 9151 raft_consensus.cc:686] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:09.259914 9151 consensus_queue.cc:237] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:09.262127 6744 catalog_manager.cc:5484] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:09.438690 8589 raft_consensus.cc:482] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:09.438849 8589 raft_consensus.cc:504] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.439294 8589 leader_election.cc:288] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:09.439746 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbaf13cbb474734a961664ae8ebe779" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:09.439918 7278 raft_consensus.cc:2454] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:09.439873 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbaf13cbb474734a961664ae8ebe779" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:09.440032 7020 raft_consensus.cc:2454] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:09.440312 6825 leader_election.cc:302] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:09.440539 8589 raft_consensus.cc:2781] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:09.440613 8589 raft_consensus.cc:482] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:09.440654 8589 raft_consensus.cc:3037] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:09.441670 8589 raft_consensus.cc:504] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.442099 8589 leader_election.cc:288] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:09.442585 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbaf13cbb474734a961664ae8ebe779" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:09.442582 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbaf13cbb474734a961664ae8ebe779" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:09.442731 7278 raft_consensus.cc:3037] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:09.442731 7020 raft_consensus.cc:3037] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:09.443832 7020 raft_consensus.cc:2454] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:09.443832 7278 raft_consensus.cc:2454] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:09.444227 6825 leader_election.cc:302] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:09.444443 8589 raft_consensus.cc:2781] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:09.444548 8589 raft_consensus.cc:686] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:09.444880 8589 consensus_queue.cc:237] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:09.447371 6744 catalog_manager.cc:5484] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
W20240503 07:59:09.598693 9229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:09.598997 9229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:09.602515 9229 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240503 07:59:09.699817 9053 consensus_queue.cc:1035] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.703368 7020 raft_consensus.cc:1261] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.703374 7278 raft_consensus.cc:1261] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.703804 8589 consensus_queue.cc:1035] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.704006 8192 consensus_queue.cc:1035] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.709452 6889 raft_consensus.cc:1261] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.709887 9053 consensus_queue.cc:1035] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.715235 7150 raft_consensus.cc:1261] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.715783 9151 consensus_queue.cc:1035] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.718689 7282 raft_consensus.cc:1261] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.719810 9151 consensus_queue.cc:1035] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.723595 9261 mvcc.cc:205] Tried to move back new op lower bound from 7023506021205405696 to 7023506019292057600. Current Snapshot: MvccSnapshot[applied={T|T < 7023506021205405696}]
INSERT report
rows total: 2000
time total: 130.515 ms
time per row: 0.0652573 ms
I20240503 07:59:09.786170 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:37815
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:44703
--webserver_port=42785
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 07:59:09.915786 9292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:09.916060 9292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:09.916126 9292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:09.919512 9292 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:09.919608 9292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:09.919711 9292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:09.924175 9292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:37815
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=42785
--tserver_master_addrs=127.0.99.190:44703
--never_fsync=true
--heap_profile_path=/tmp/kudu.9292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:09.925233 9292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:09.926530 9292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:09.936218 9297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:09.936635 9298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:09.937173 9300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:09.937916 9292 server_base.cc:1034] running on GCE node
I20240503 07:59:09.938500 9292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:09.939214 9292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:09.940369 9292 hybrid_clock.cc:648] HybridClock initialized: now 1714723149940332 us; error 55 us; skew 500 ppm
I20240503 07:59:09.942811 9292 webserver.cc:460] Webserver started at http://127.0.99.133:42785/ using document root <none> and password file <none>
I20240503 07:59:09.943531 9292 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:09.943631 9292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:09.947499 9292 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240503 07:59:09.949661 9306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.950974 9292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:09.951125 9292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3befbb2d206f4be89d3e07f38837374b"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:59:09.951737 9292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:09.982151 9292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:09.983085 9292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:09.983289 9292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:09.984005 9292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:09.985196 9292 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:09.985267 9292 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.985317 9292 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:09.985342 9292 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.995628 9292 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:37815
I20240503 07:59:09.995831 9419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:37815 every 8 connection(s)
I20240503 07:59:09.997071 9292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:10.003420 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 9292
I20240503 07:59:10.003870 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:45435
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:44703
--webserver_port=41199
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:10.009009 9420 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44703
I20240503 07:59:10.009279 9420 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:10.009902 9420 heartbeater.cc:511] Master 127.0.99.190:44703 requested a full tablet report, sending...
I20240503 07:59:10.011279 6745 ts_manager.cc:194] Re-registered known tserver with Master: 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:10.012025 6745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:58687
W20240503 07:59:10.136441 9423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:10.136701 9423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:10.136749 9423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:10.140197 9423 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:10.140275 9423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:10.140369 9423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:59:10.144716 9423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:45435
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=41199
--tserver_master_addrs=127.0.99.190:44703
--never_fsync=true
--heap_profile_path=/tmp/kudu.9423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:10.145759 9423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:10.147070 9423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:10.156513 9429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.157029 9431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.157584 9428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:10.158143 9423 server_base.cc:1034] running on GCE node
I20240503 07:59:10.158671 9423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:10.159360 9423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:10.160511 9423 hybrid_clock.cc:648] HybridClock initialized: now 1714723150160480 us; error 44 us; skew 500 ppm
I20240503 07:59:10.162623 9423 webserver.cc:460] Webserver started at http://127.0.99.134:41199/ using document root <none> and password file <none>
I20240503 07:59:10.163251 9423 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:10.163332 9423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:10.167408 9423 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:10.169602 9437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.170876 9423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:10.171010 9423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "cb73994515af4880b05383e78368bb4a"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-bj8b"
I20240503 07:59:10.171542 9423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:10.201674 9423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:10.202440 9423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:10.202616 9423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:10.203213 9423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:10.204172 9423 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:10.204221 9423 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.204262 9423 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:10.204275 9423 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.213668 9423 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:45435
I20240503 07:59:10.213683 9550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:45435 every 8 connection(s)
I20240503 07:59:10.215023 9423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:59:10.221606 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 9423
I20240503 07:59:10.222023 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:43369
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:44703
--webserver_port=42821
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:10.224751 9551 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44703
I20240503 07:59:10.225018 9551 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:10.225562 9551 heartbeater.cc:511] Master 127.0.99.190:44703 requested a full tablet report, sending...
I20240503 07:59:10.226696 6745 ts_manager.cc:194] Re-registered known tserver with Master: cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:10.227320 6745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:39873
W20240503 07:59:10.349617 9554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:10.349895 9554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:10.349956 9554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:10.353245 9554 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:10.353331 9554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:10.353427 9554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:59:10.357723 9554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:43369
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=42821
--tserver_master_addrs=127.0.99.190:44703
--never_fsync=true
--heap_profile_path=/tmp/kudu.9554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:10.359002 9554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:10.360383 9554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:10.370265 9560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.370748 9562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.371448 9559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:10.371872 9554 server_base.cc:1034] running on GCE node
I20240503 07:59:10.372321 9554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:10.373026 9554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:10.374217 9554 hybrid_clock.cc:648] HybridClock initialized: now 1714723150374150 us; error 80 us; skew 500 ppm
I20240503 07:59:10.376533 9554 webserver.cc:460] Webserver started at http://127.0.99.135:42821/ using document root <none> and password file <none>
I20240503 07:59:10.377197 9554 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:10.377274 9554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:10.381562 9554 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:10.383966 9568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.385265 9554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:10.385385 9554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "159177b3caeb434baa98e04136d050b2"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-bj8b"
I20240503 07:59:10.385969 9554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:10.407176 9554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:10.407909 9554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:10.408079 9554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:10.408805 9554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:10.409957 9554 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:10.410012 9554 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.410073 9554 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:10.410094 9554 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.419524 9554 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:43369
I20240503 07:59:10.419545 9681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:43369 every 8 connection(s)
I20240503 07:59:10.420720 9554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:59:10.428791 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 9554
I20240503 07:59:10.429306 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
/tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:43683
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:44703
--webserver_port=43201
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:10.431087 9682 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44703
I20240503 07:59:10.431346 9682 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:10.431952 9682 heartbeater.cc:511] Master 127.0.99.190:44703 requested a full tablet report, sending...
I20240503 07:59:10.433145 6745 ts_manager.cc:194] Re-registered known tserver with Master: 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:10.433830 6745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:59053
W20240503 07:59:10.556241 9685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:10.556494 9685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:10.556558 9685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:10.559950 9685 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:10.560041 9685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:10.560148 9685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:59:10.564210 9685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:43683
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=43201
--tserver_master_addrs=127.0.99.190:44703
--never_fsync=true
--heap_profile_path=/tmp/kudu.9685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:10.565304 9685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:10.566596 9685 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:10.576112 9690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.576637 9691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.577229 9693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:10.577860 9685 server_base.cc:1034] running on GCE node
I20240503 07:59:10.578377 9685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:10.579095 9685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:10.580283 9685 hybrid_clock.cc:648] HybridClock initialized: now 1714723150580267 us; error 27 us; skew 500 ppm
I20240503 07:59:10.582827 9685 webserver.cc:460] Webserver started at http://127.0.99.136:43201/ using document root <none> and password file <none>
I20240503 07:59:10.583549 9685 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:10.583640 9685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:10.588554 9685 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 07:59:10.591346 9699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.592664 9685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:10.592793 9685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a372e98a670f49e9adca46ab5c836c4a"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-bj8b"
I20240503 07:59:10.593278 9685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:10.613296 9685 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:10.614074 9685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:10.614277 9685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:10.615056 9685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:10.616122 9685 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:10.616170 9685 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.616211 9685 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:10.616232 9685 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.625227 9685 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:43683
I20240503 07:59:10.625334 9812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:43683 every 8 connection(s)
I20240503 07:59:10.626451 9685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:59:10.636571 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ1Z_mA/build/debug/bin/kudu as pid 9685
I20240503 07:59:10.636788 9813 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44703
I20240503 07:59:10.637055 9813 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:10.637676 9813 heartbeater.cc:511] Master 127.0.99.190:44703 requested a full tablet report, sending...
I20240503 07:59:10.638840 6745 ts_manager.cc:194] Re-registered known tserver with Master: a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:10.639626 6745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:58173
I20240503 07:59:10.639840 6744 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:10.646427 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8b559fc5214070ab822e5b36391179 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec052d72df7240578edb2054572c36fe]), partition=
I20240503 07:59:10.646788 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8b559fc5214070ab822e5b36391179. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.649029 9829 tablet_bootstrap.cc:492] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:10.649889 9829 tablet_bootstrap.cc:654] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.651099 9829 tablet_bootstrap.cc:492] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:10.651201 9829 ts_tablet_manager.cc:1367] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:10.651669 9829 raft_consensus.cc:348] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:10.651790 9829 raft_consensus.cc:374] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.651820 9829 raft_consensus.cc:729] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.651953 9829 consensus_queue.cc:260] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:10.652240 9829 ts_tablet_manager.cc:1398] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:10.657121 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8b559fc5214070ab822e5b36391179 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec052d72df7240578edb2054572c36fe]), partition=
I20240503 07:59:10.658017 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8b559fc5214070ab822e5b36391179. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.660487 8192 raft_consensus.cc:482] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:10.660600 8192 raft_consensus.cc:504] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:10.661157 8192 leader_election.cc:288] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:10.663328 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8b559fc5214070ab822e5b36391179 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec052d72df7240578edb2054572c36fe]), partition=
I20240503 07:59:10.664166 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8b559fc5214070ab822e5b36391179. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 07:59:10.670083 6822 leader_election.cc:341] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Not found: Tablet not found: 4d8b559fc5214070ab822e5b36391179
W20240503 07:59:10.670261 6825 leader_election.cc:341] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Not found: Tablet not found: 4d8b559fc5214070ab822e5b36391179
I20240503 07:59:10.670333 6825 leader_election.cc:302] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92; no voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a
I20240503 07:59:10.670678 8192 raft_consensus.cc:2726] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:10.675433 9833 tablet_bootstrap.cc:492] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:10.680166 9834 tablet_bootstrap.cc:492] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:10.681643 9833 tablet_bootstrap.cc:654] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.682612 9834 tablet_bootstrap.cc:654] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.682624 9833 log.cc:824] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:10.683416 9834 log.cc:824] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:10.685268 9834 tablet_bootstrap.cc:492] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:10.685488 9834 ts_tablet_manager.cc:1367] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 07:59:10.685647 9833 tablet_bootstrap.cc:492] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:10.685832 9833 ts_tablet_manager.cc:1367] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 07:59:10.689568 9833 raft_consensus.cc:348] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:10.689568 9834 raft_consensus.cc:348] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:10.689821 9833 raft_consensus.cc:374] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.689821 9834 raft_consensus.cc:374] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.689879 9834 raft_consensus.cc:729] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.689877 9833 raft_consensus.cc:729] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.690428 9834 consensus_queue.cc:260] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:10.690510 9833 consensus_queue.cc:260] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:10.691454 9834 ts_tablet_manager.cc:1398] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 07:59:10.691454 9833 ts_tablet_manager.cc:1398] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 07:59:10.691890 9682 heartbeater.cc:503] Master 127.0.99.190:44703 was elected leader, sending a full tablet report...
I20240503 07:59:10.692917 9551 heartbeater.cc:503] Master 127.0.99.190:44703 was elected leader, sending a full tablet report...
W20240503 07:59:10.716229 9552 tablet.cc:2369] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:10.841897 9837 raft_consensus.cc:482] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:10.842120 9837 raft_consensus.cc:504] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:10.843730 9837 leader_election.cc:288] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:10.848487 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8b559fc5214070ab822e5b36391179" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:10.848853 9505 raft_consensus.cc:2454] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:10.849416 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8b559fc5214070ab822e5b36391179" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:10.849586 6888 raft_consensus.cc:2454] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:10.849598 9572 leader_election.cc:302] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:10.850235 9837 raft_consensus.cc:2781] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:10.850319 9837 raft_consensus.cc:482] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:10.850391 9837 raft_consensus.cc:3037] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.851625 9837 raft_consensus.cc:504] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:10.852414 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8b559fc5214070ab822e5b36391179" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:10.852560 6888 raft_consensus.cc:3037] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.852665 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8b559fc5214070ab822e5b36391179" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:10.852833 9505 raft_consensus.cc:3037] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.853767 6888 raft_consensus.cc:2454] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:10.854038 9505 raft_consensus.cc:2454] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:10.854151 9569 leader_election.cc:302] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:10.853978 9837 leader_election.cc:288] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:10.854635 9837 raft_consensus.cc:2781] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:10.854722 9837 raft_consensus.cc:686] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:10.855059 9837 consensus_queue.cc:237] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:10.857969 6741 catalog_manager.cc:5484] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
W20240503 07:59:10.921911 9683 tablet.cc:2369] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:10.937216 6999 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:10.937927 9616 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:10.938530 6869 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:10.942894 7129 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:10.943475 7260 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:10.946302 9350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:10.947710 9747 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:10.979694 9485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:10.989662 6747 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:10.997850 6888 raft_consensus.cc:1261] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:10.997897 9505 raft_consensus.cc:1261] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:10.998812 9837 consensus_queue.cc:1035] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.003615 9837 consensus_queue.cc:1035] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.005039 6738 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 4d8b559fc5214070ab822e5b36391179 (table rebalancer_dynamic_table [id=ec052d72df7240578edb2054572c36fe]). Expected version 1 got 0
I20240503 07:59:11.008744 6738 catalog_manager.cc:5576] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655) does not have the latest schema for tablet 4d8b559fc5214070ab822e5b36391179 (table rebalancer_dynamic_table [id=ec052d72df7240578edb2054572c36fe]). Expected version 1 got 0
I20240503 07:59:11.014238 9420 heartbeater.cc:503] Master 127.0.99.190:44703 was elected leader, sending a full tablet report...
I20240503 07:59:11.013764 9906 tablet.cc:1722] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:11.015344 9906 tablet.cc:1620] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:11.015581 6738 catalog_manager.cc:5576] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655) does not have the latest schema for tablet 4d8b559fc5214070ab822e5b36391179 (table rebalancer_dynamic_table [id=ec052d72df7240578edb2054572c36fe]). Expected version 1 got 0
I20240503 07:59:11.015731 9901 mvcc.cc:205] Tried to move back new op lower bound from 7023506026479210496 to 7023506025905377280. Current Snapshot: MvccSnapshot[applied={T|T < 7023506026479210496}]
I20240503 07:59:11.016003 9907 tablet.cc:1722] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:11.017189 9907 tablet.cc:1620] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:11.020637 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=ec052d72df7240578edb2054572c36fe] alter complete (version 1)
I20240503 07:59:11.029778 9908 tablet.cc:1722] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:11.030849 9908 tablet.cc:1620] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:11.031029 6738 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 4d8b559fc5214070ab822e5b36391179 (table rebalancer_dynamic_table [id=ec052d72df7240578edb2054572c36fe]). Expected version 1 got 0
I20240503 07:59:11.124336 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:11.130983 9906 tablet.cc:1722] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:11.131383 9906 tablet.cc:1620] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:11.131484 9907 tablet.cc:1722] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:11.131768 9908 tablet.cc:1722] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:11.131809 9907 tablet.cc:1620] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:11.132054 9908 tablet.cc:1620] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:11.137164 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=ec052d72df7240578edb2054572c36fe] alter complete (version 2)
I20240503 07:59:11.139508 6743 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:11.139948 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 07:59:11.140090 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 07:59:11.143877 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 0a53d599d057465496a555820cd6742d
I20240503 07:59:11.144722 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a53d599d057465496a555820cd6742d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:11.144811 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 24bfdc19078046259a38df8e61fdaee5
I20240503 07:59:11.145279 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a53d599d057465496a555820cd6742d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:11.145758 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a53d599d057465496a555820cd6742d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:11.145808 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 779675bcd7d7497ebe50cbb1875fb1e7
I20240503 07:59:11.145921 7130 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bfdc19078046259a38df8e61fdaee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:11.146512 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 779675bcd7d7497ebe50cbb1875fb1e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:11.147202 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 779675bcd7d7497ebe50cbb1875fb1e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:11.147317 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bfdc19078046259a38df8e61fdaee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:11.147972 9957 tablet_replica.cc:330] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:11.148365 9957 raft_consensus.cc:2229] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.148808 9957 raft_consensus.cc:2258] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.149731 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bfdc19078046259a38df8e61fdaee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:11.150095 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet 779675bcd7d7497ebe50cbb1875fb1e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:11.149945 9958 tablet_replica.cc:330] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:11.150584 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 50169f7d514d4fe3a4f846faaed03325 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=62034a44b5d9412f853d552a90206f04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.150782 9958 raft_consensus.cc:2229] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.150907 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50169f7d514d4fe3a4f846faaed03325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.151515 9958 raft_consensus.cc:2258] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.153084 9957 ts_tablet_manager.cc:1875] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.153313 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 23759e1c50b9402980461599687904b6 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=62034a44b5d9412f853d552a90206f04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.153591 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 23759e1c50b9402980461599687904b6 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=62034a44b5d9412f853d552a90206f04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.153718 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23759e1c50b9402980461599687904b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.153813 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23759e1c50b9402980461599687904b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.157032 9833 tablet_bootstrap.cc:492] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:11.157884 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 50169f7d514d4fe3a4f846faaed03325 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=62034a44b5d9412f853d552a90206f04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.158017 9833 tablet_bootstrap.cc:654] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.158180 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50169f7d514d4fe3a4f846faaed03325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.158003 9484 tablet_service.cc:1467] Processing CreateTablet for tablet dae94f108f8142379c271e9fc22524ef (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=62034a44b5d9412f853d552a90206f04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.158442 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae94f108f8142379c271e9fc22524ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.160902 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 23759e1c50b9402980461599687904b6 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=62034a44b5d9412f853d552a90206f04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.161165 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23759e1c50b9402980461599687904b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.161114 9350 tablet_service.cc:1467] Processing CreateTablet for tablet dae94f108f8142379c271e9fc22524ef (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=62034a44b5d9412f853d552a90206f04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.162205 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae94f108f8142379c271e9fc22524ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.162494 9834 tablet_bootstrap.cc:492] T dae94f108f8142379c271e9fc22524ef P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:11.163436 9834 tablet_bootstrap.cc:654] T dae94f108f8142379c271e9fc22524ef P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.164434 9957 ts_tablet_manager.cc:1888] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.164611 9957 log.cc:1196] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0a53d599d057465496a555820cd6742d
I20240503 07:59:11.164541 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 50169f7d514d4fe3a4f846faaed03325 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=62034a44b5d9412f853d552a90206f04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.165153 9957 ts_tablet_manager.cc:1909] T 0a53d599d057465496a555820cd6742d P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:11.165694 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50169f7d514d4fe3a4f846faaed03325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.166479 9957 tablet_replica.cc:330] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:11.166770 9957 raft_consensus.cc:2229] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.167519 9746 tablet_service.cc:1467] Processing CreateTablet for tablet dae94f108f8142379c271e9fc22524ef (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=62034a44b5d9412f853d552a90206f04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.167841 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae94f108f8142379c271e9fc22524ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.168749 9958 ts_tablet_manager.cc:1875] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.169620 9957 raft_consensus.cc:2258] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.170490 9834 tablet_bootstrap.cc:492] T dae94f108f8142379c271e9fc22524ef P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:11.170591 9834 ts_tablet_manager.cc:1367] T dae94f108f8142379c271e9fc22524ef P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:11.170629 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 0a53d599d057465496a555820cd6742d (table rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]) successfully deleted
I20240503 07:59:11.171391 9834 raft_consensus.cc:348] T dae94f108f8142379c271e9fc22524ef P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:11.171168 9961 tablet_bootstrap.cc:492] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:11.171522 9834 raft_consensus.cc:374] T dae94f108f8142379c271e9fc22524ef P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.171559 9834 raft_consensus.cc:729] T dae94f108f8142379c271e9fc22524ef P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.171682 9834 consensus_queue.cc:260] T dae94f108f8142379c271e9fc22524ef P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:11.171967 9834 ts_tablet_manager.cc:1398] T dae94f108f8142379c271e9fc22524ef P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.173115 9957 ts_tablet_manager.cc:1875] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.173254 9834 tablet_bootstrap.cc:492] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:11.174427 9961 tablet_bootstrap.cc:654] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.175235 9956 tablet_replica.cc:330] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:11.175788 9956 raft_consensus.cc:2229] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.176229 9956 raft_consensus.cc:2258] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.176457 9957 ts_tablet_manager.cc:1888] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.176592 9957 log.cc:1196] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/24bfdc19078046259a38df8e61fdaee5
I20240503 07:59:11.176934 9957 ts_tablet_manager.cc:1909] T 24bfdc19078046259a38df8e61fdaee5 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:11.177482 9956 ts_tablet_manager.cc:1875] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.178393 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 24bfdc19078046259a38df8e61fdaee5 (table rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]) successfully deleted
I20240503 07:59:11.178637 9957 tablet_replica.cc:330] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:11.178687 9959 tablet_replica.cc:330] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:11.178836 9957 raft_consensus.cc:2229] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.179404 9957 raft_consensus.cc:2258] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.179389 9959 raft_consensus.cc:2229] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.179595 9959 raft_consensus.cc:2258] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.180002 9957 ts_tablet_manager.cc:1875] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.181284 9834 tablet_bootstrap.cc:654] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.182634 9959 ts_tablet_manager.cc:1875] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.182996 9965 tablet_bootstrap.cc:492] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:11.183607 9957 ts_tablet_manager.cc:1888] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.183743 9957 log.cc:1196] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/779675bcd7d7497ebe50cbb1875fb1e7
I20240503 07:59:11.184063 9957 ts_tablet_manager.cc:1909] T 779675bcd7d7497ebe50cbb1875fb1e7 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:11.182657 9960 tablet_bootstrap.cc:492] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:11.185405 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 779675bcd7d7497ebe50cbb1875fb1e7 (table rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]) successfully deleted
I20240503 07:59:11.185532 9965 tablet_bootstrap.cc:654] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.185804 9960 tablet_bootstrap.cc:654] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.186313 9965 log.cc:824] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:11.186524 9834 tablet_bootstrap.cc:492] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:11.186630 9834 ts_tablet_manager.cc:1367] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:11.187117 9834 raft_consensus.cc:348] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:11.187239 9834 raft_consensus.cc:374] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.187269 9834 raft_consensus.cc:729] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.187404 9834 consensus_queue.cc:260] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:11.187695 9834 ts_tablet_manager.cc:1398] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:11.188148 9833 tablet_bootstrap.cc:492] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:11.188234 9833 ts_tablet_manager.cc:1367] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240503 07:59:11.188338 9965 tablet_bootstrap.cc:492] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:11.188410 9956 ts_tablet_manager.cc:1888] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.188504 9956 log.cc:1196] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0a53d599d057465496a555820cd6742d
I20240503 07:59:11.188531 9965 ts_tablet_manager.cc:1367] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 07:59:11.188673 9833 raft_consensus.cc:348] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:11.188755 9833 raft_consensus.cc:374] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.188778 9833 raft_consensus.cc:729] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.188828 9956 ts_tablet_manager.cc:1909] T 0a53d599d057465496a555820cd6742d P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:11.188894 9833 consensus_queue.cc:260] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:11.189142 9833 ts_tablet_manager.cc:1398] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:11.190332 9958 ts_tablet_manager.cc:1888] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.190459 9958 log.cc:1196] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0a53d599d057465496a555820cd6742d
I20240503 07:59:11.190490 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 0a53d599d057465496a555820cd6742d (table rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]) successfully deleted
I20240503 07:59:11.190784 9958 ts_tablet_manager.cc:1909] T 0a53d599d057465496a555820cd6742d P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:11.190831 9956 tablet_replica.cc:330] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:11.191005 9956 raft_consensus.cc:2229] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.192651 9959 ts_tablet_manager.cc:1888] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.192804 9959 log.cc:1196] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/24bfdc19078046259a38df8e61fdaee5
I20240503 07:59:11.193272 9959 ts_tablet_manager.cc:1909] T 24bfdc19078046259a38df8e61fdaee5 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:11.193176 9965 raft_consensus.cc:348] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:11.194480 9965 raft_consensus.cc:374] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.194552 9965 raft_consensus.cc:729] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.194823 9959 tablet_replica.cc:330] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:11.194976 9959 raft_consensus.cc:2229] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.195097 9959 raft_consensus.cc:2258] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.195139 9965 consensus_queue.cc:260] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:11.195645 9959 ts_tablet_manager.cc:1875] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.191809 9961 tablet_bootstrap.cc:492] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:11.195803 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 24bfdc19078046259a38df8e61fdaee5 (table rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]) successfully deleted
I20240503 07:59:11.195868 9956 raft_consensus.cc:2258] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.195972 9961 ts_tablet_manager.cc:1367] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 07:59:11.196138 9965 ts_tablet_manager.cc:1398] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 07:59:11.196432 9956 ts_tablet_manager.cc:1875] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.196532 9813 heartbeater.cc:503] Master 127.0.99.190:44703 was elected leader, sending a full tablet report...
I20240503 07:59:11.197335 9961 raft_consensus.cc:348] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:11.197558 9961 raft_consensus.cc:374] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.197633 9961 raft_consensus.cc:729] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.197862 9965 tablet_bootstrap.cc:492] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:11.199067 9960 tablet_bootstrap.cc:492] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:11.198804 9961 consensus_queue.cc:260] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:11.199335 9965 tablet_bootstrap.cc:654] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.199420 9961 ts_tablet_manager.cc:1398] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:11.200148 9960 ts_tablet_manager.cc:1367] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 07:59:11.200500 9965 tablet_bootstrap.cc:492] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:11.200587 9965 ts_tablet_manager.cc:1367] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:11.200783 9960 raft_consensus.cc:348] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:11.200966 9960 raft_consensus.cc:374] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.201043 9960 raft_consensus.cc:729] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.201139 9965 raft_consensus.cc:348] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:11.201251 9965 raft_consensus.cc:374] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.201285 9965 raft_consensus.cc:729] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.201406 9960 consensus_queue.cc:260] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:11.201529 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 0a53d599d057465496a555820cd6742d (table rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]) successfully deleted
I20240503 07:59:11.201854 9960 ts_tablet_manager.cc:1398] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:11.202064 9960 tablet_bootstrap.cc:492] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:11.202653 9979 tablet_bootstrap.cc:492] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:11.202682 9958 tablet_replica.cc:330] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:11.202955 9965 consensus_queue.cc:260] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:11.203503 9965 ts_tablet_manager.cc:1398] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:11.204110 9960 tablet_bootstrap.cc:654] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.202843 9958 raft_consensus.cc:2229] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.204706 9958 raft_consensus.cc:2258] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.206024 9958 ts_tablet_manager.cc:1875] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.207650 9979 tablet_bootstrap.cc:654] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.208647 9979 log.cc:824] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:11.209442 9960 tablet_bootstrap.cc:492] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:11.209594 9960 ts_tablet_manager.cc:1367] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:11.209985 9956 ts_tablet_manager.cc:1888] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.210081 9960 raft_consensus.cc:348] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:11.210678 9960 raft_consensus.cc:374] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.210768 9960 raft_consensus.cc:729] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.210961 9960 consensus_queue.cc:260] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:11.211491 9960 ts_tablet_manager.cc:1398] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:11.213117 9956 log.cc:1196] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/779675bcd7d7497ebe50cbb1875fb1e7
I20240503 07:59:11.213407 9979 tablet_bootstrap.cc:492] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:11.213621 9979 ts_tablet_manager.cc:1367] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240503 07:59:11.215204 9958 ts_tablet_manager.cc:1888] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.215359 9958 log.cc:1196] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/24bfdc19078046259a38df8e61fdaee5
I20240503 07:59:11.215453 9956 ts_tablet_manager.cc:1909] T 779675bcd7d7497ebe50cbb1875fb1e7 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:11.215675 9958 ts_tablet_manager.cc:1909] T 24bfdc19078046259a38df8e61fdaee5 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:11.216471 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 24bfdc19078046259a38df8e61fdaee5 (table rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]) successfully deleted
I20240503 07:59:11.216737 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 779675bcd7d7497ebe50cbb1875fb1e7 (table rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]) successfully deleted
I20240503 07:59:11.217274 9979 raft_consensus.cc:348] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:11.217599 9979 raft_consensus.cc:374] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.217708 9979 raft_consensus.cc:729] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.218338 9979 consensus_queue.cc:260] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:11.219512 9979 ts_tablet_manager.cc:1398] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240503 07:59:11.221244 9616 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:11.223430 9959 ts_tablet_manager.cc:1888] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.236562 7129 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:11.242235 9747 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.243588 6743 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:11.250299 9959 log.cc:1196] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/779675bcd7d7497ebe50cbb1875fb1e7
I20240503 07:59:11.251433 9906 tablet.cc:1722] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:11.252432 9959 ts_tablet_manager.cc:1909] T 779675bcd7d7497ebe50cbb1875fb1e7 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
W20240503 07:59:11.252753 9421 tablet.cc:2369] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:11.253410 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 779675bcd7d7497ebe50cbb1875fb1e7 (table rebalance_test_table_0 [id=89abe38dc7fc436f94137b7d60a3fefc]) successfully deleted
I20240503 07:59:11.255620 9485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.256525 9907 tablet.cc:1722] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:11.259388 9908 tablet.cc:1722] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:11.261188 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:11.265870 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=ec052d72df7240578edb2054572c36fe] alter complete (version 3)
I20240503 07:59:11.267031 6869 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:11.287377 9350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.288707 7260 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:11.324105 9977 raft_consensus.cc:482] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.324344 9977 raft_consensus.cc:504] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:11.325769 9977 leader_election.cc:288] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:11.331395 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae94f108f8142379c271e9fc22524ef" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:11.331570 9505 raft_consensus.cc:2454] T dae94f108f8142379c271e9fc22524ef P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:11.332152 9703 leader_election.cc:302] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:11.332471 9977 raft_consensus.cc:2781] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:11.332535 9977 raft_consensus.cc:482] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:11.332569 9977 raft_consensus.cc:3037] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.333724 9977 raft_consensus.cc:504] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:11.334160 9977 leader_election.cc:288] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:11.334969 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae94f108f8142379c271e9fc22524ef" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:11.335107 9505 raft_consensus.cc:3037] T dae94f108f8142379c271e9fc22524ef P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.336148 9505 raft_consensus.cc:2454] T dae94f108f8142379c271e9fc22524ef P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:11.336107 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae94f108f8142379c271e9fc22524ef" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:11.336428 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae94f108f8142379c271e9fc22524ef" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:11.336473 9357 raft_consensus.cc:2454] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:11.337221 9703 leader_election.cc:302] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:11.337482 9977 raft_consensus.cc:2781] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:11.337693 9977 raft_consensus.cc:686] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:11.338057 9977 consensus_queue.cc:237] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:11.342273 6739 catalog_manager.cc:5484] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:11.370951 6739 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:11.371129 6739 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:11.374392 6739 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 4d8b559fc5214070ab822e5b36391179
I20240503 07:59:11.375458 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8b559fc5214070ab822e5b36391179 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:11.375494 9616 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8b559fc5214070ab822e5b36391179 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:11.375694 9957 tablet_replica.cc:330] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:11.375855 9957 raft_consensus.cc:2229] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.376008 9957 raft_consensus.cc:2258] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.376941 9957 ts_tablet_manager.cc:1875] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 07:59:11.378181 9814 tablet.cc:2369] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:11.379221 9957 ts_tablet_manager.cc:1888] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.379302 9957 log.cc:1196] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4d8b559fc5214070ab822e5b36391179
I20240503 07:59:11.379547 9957 ts_tablet_manager.cc:1909] T 4d8b559fc5214070ab822e5b36391179 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:11.380414 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 4d8b559fc5214070ab822e5b36391179 (table rebalancer_dynamic_table_new_name [id=ec052d72df7240578edb2054572c36fe]) successfully deleted
I20240503 07:59:11.381065 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8b559fc5214070ab822e5b36391179 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:11.381542 10006 tablet_replica.cc:330] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:11.381973 10006 raft_consensus.cc:2229] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.382460 10006 raft_consensus.cc:2258] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.383287 10005 tablet_replica.cc:330] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:11.383731 10005 raft_consensus.cc:2229] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.383893 10006 ts_tablet_manager.cc:1875] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.384178 10005 raft_consensus.cc:2258] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.385380 10005 ts_tablet_manager.cc:1875] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.386479 10006 ts_tablet_manager.cc:1888] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.386641 10006 log.cc:1196] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/4d8b559fc5214070ab822e5b36391179
I20240503 07:59:11.387019 10006 ts_tablet_manager.cc:1909] T 4d8b559fc5214070ab822e5b36391179 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:11.387878 10005 ts_tablet_manager.cc:1888] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.387974 10005 log.cc:1196] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/4d8b559fc5214070ab822e5b36391179
I20240503 07:59:11.388264 10005 ts_tablet_manager.cc:1909] T 4d8b559fc5214070ab822e5b36391179 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:11.388754 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 4d8b559fc5214070ab822e5b36391179 (table rebalancer_dynamic_table_new_name [id=ec052d72df7240578edb2054572c36fe]) successfully deleted
I20240503 07:59:11.389393 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 4d8b559fc5214070ab822e5b36391179 (table rebalancer_dynamic_table_new_name [id=ec052d72df7240578edb2054572c36fe]) successfully deleted
I20240503 07:59:11.477619 6747 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:11.485718 9747 tablet_service.cc:1467] Processing CreateTablet for tablet b6a74bf652c7452b9665126a17cfad47 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0261b84d392944268c2d8ef3e031ebb7]), partition=
I20240503 07:59:11.486032 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a74bf652c7452b9665126a17cfad47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.489207 9350 tablet_service.cc:1467] Processing CreateTablet for tablet b6a74bf652c7452b9665126a17cfad47 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0261b84d392944268c2d8ef3e031ebb7]), partition=
I20240503 07:59:11.489507 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a74bf652c7452b9665126a17cfad47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.492403 9979 tablet_bootstrap.cc:492] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:11.493391 9979 tablet_bootstrap.cc:654] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.494207 9965 tablet_bootstrap.cc:492] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:11.495348 9965 tablet_bootstrap.cc:654] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.496004 9881 raft_consensus.cc:482] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.496116 9881 raft_consensus.cc:504] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:11.496685 9881 leader_election.cc:288] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:11.497213 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23759e1c50b9402980461599687904b6" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:11.497390 9505 raft_consensus.cc:2454] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:11.497892 7260 tablet_service.cc:1467] Processing CreateTablet for tablet b6a74bf652c7452b9665126a17cfad47 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0261b84d392944268c2d8ef3e031ebb7]), partition=
I20240503 07:59:11.498174 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a74bf652c7452b9665126a17cfad47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.500205 9979 tablet_bootstrap.cc:492] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:11.500326 9979 ts_tablet_manager.cc:1367] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:11.500584 9961 tablet_bootstrap.cc:492] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:11.500726 9572 leader_election.cc:302] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:11.500994 9881 raft_consensus.cc:2781] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:11.501045 9881 raft_consensus.cc:482] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:11.500968 9979 raft_consensus.cc:348] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:11.501080 9881 raft_consensus.cc:3037] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.501087 9979 raft_consensus.cc:374] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.501119 9979 raft_consensus.cc:729] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.501289 9979 consensus_queue.cc:260] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:11.501471 9961 tablet_bootstrap.cc:654] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.502034 9053 raft_consensus.cc:482] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.502115 9053 raft_consensus.cc:504] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:11.502137 9881 raft_consensus.cc:504] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:11.502534 9881 leader_election.cc:288] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:11.502628 9979 ts_tablet_manager.cc:1398] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:11.503249 9053 leader_election.cc:288] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:11.503329 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23759e1c50b9402980461599687904b6" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:11.503460 9505 raft_consensus.cc:3037] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.504482 9505 raft_consensus.cc:2454] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:11.505110 9572 leader_election.cc:302] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:11.505326 9881 raft_consensus.cc:2781] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:11.505402 9881 raft_consensus.cc:686] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:11.505545 9881 consensus_queue.cc:237] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:11.505693 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50169f7d514d4fe3a4f846faaed03325" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:11.505851 7020 raft_consensus.cc:2454] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:11.506095 9961 tablet_bootstrap.cc:492] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:11.506186 9961 ts_tablet_manager.cc:1367] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:11.506651 9961 raft_consensus.cc:348] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:11.506754 9961 raft_consensus.cc:374] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.506783 9961 raft_consensus.cc:729] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.506901 9961 consensus_queue.cc:260] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:11.507194 9961 ts_tablet_manager.cc:1398] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.508100 7217 leader_election.cc:302] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:11.508347 6743 catalog_manager.cc:5484] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:11.508829 9053 raft_consensus.cc:2781] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:11.508883 9053 raft_consensus.cc:482] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:11.508919 9053 raft_consensus.cc:3037] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.509994 9053 raft_consensus.cc:504] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:11.511121 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23759e1c50b9402980461599687904b6" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:11.511258 7018 raft_consensus.cc:3037] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.512550 7018 raft_consensus.cc:2454] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:11.513633 9053 leader_election.cc:288] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:11.514097 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50169f7d514d4fe3a4f846faaed03325" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:11.514254 7018 raft_consensus.cc:3037] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.515031 7018 raft_consensus.cc:2454] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:11.515343 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23759e1c50b9402980461599687904b6" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:11.515419 7217 leader_election.cc:302] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:11.515486 7020 raft_consensus.cc:2362] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1. Re-sending same reply.
I20240503 07:59:11.515630 9053 raft_consensus.cc:2781] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:11.515736 9053 raft_consensus.cc:686] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:11.515887 9053 consensus_queue.cc:237] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:11.518401 6741 catalog_manager.cc:5484] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:11.520172 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50169f7d514d4fe3a4f846faaed03325" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:11.520467 9767 raft_consensus.cc:2454] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:11.520884 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50169f7d514d4fe3a4f846faaed03325" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:11.520995 9766 raft_consensus.cc:3037] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.521953 9965 tablet_bootstrap.cc:492] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:11.522047 9965 ts_tablet_manager.cc:1367] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240503 07:59:11.522150 9766 raft_consensus.cc:2454] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:11.522596 9965 raft_consensus.cc:348] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:11.522708 9965 raft_consensus.cc:374] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.522738 9965 raft_consensus.cc:729] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.522914 9965 consensus_queue.cc:260] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:11.526124 9965 ts_tablet_manager.cc:1398] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:11.545145 7129 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:11.554526 9747 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:11.562803 6869 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:11.567382 7260 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:11.571403 9616 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:11.571695 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:11.589644 9350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.602697 9485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.605626 9059 raft_consensus.cc:482] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.605765 9059 raft_consensus.cc:504] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:11.606320 9059 leader_election.cc:288] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:11.612694 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a74bf652c7452b9665126a17cfad47" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:11.612872 9766 raft_consensus.cc:2454] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:11.621877 9985 raft_consensus.cc:482] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.622033 9985 raft_consensus.cc:504] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:11.623407 9985 leader_election.cc:288] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:11.624033 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a74bf652c7452b9665126a17cfad47" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:11.624207 9374 raft_consensus.cc:2454] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:11.624596 7218 leader_election.cc:302] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:11.629029 9059 raft_consensus.cc:2781] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:11.629129 9059 raft_consensus.cc:482] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:11.629181 9059 raft_consensus.cc:3037] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.630396 9059 raft_consensus.cc:504] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:11.630844 9059 leader_election.cc:288] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:11.631752 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a74bf652c7452b9665126a17cfad47" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:11.631901 9374 raft_consensus.cc:3037] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.633114 9374 raft_consensus.cc:2454] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:11.633721 7216 leader_election.cc:302] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:11.634020 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a74bf652c7452b9665126a17cfad47" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:11.634183 9766 raft_consensus.cc:3037] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.635192 9766 raft_consensus.cc:2454] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:11.637986 9059 raft_consensus.cc:2781] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:11.638283 9059 raft_consensus.cc:686] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:11.638505 9059 consensus_queue.cc:237] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:11.639290 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a74bf652c7452b9665126a17cfad47" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:11.639488 9766 raft_consensus.cc:2379] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3befbb2d206f4be89d3e07f38837374b in current term 1: Already voted for candidate 5227b26243484bbfb177a995ceb50afc in this term.
I20240503 07:59:11.640313 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a74bf652c7452b9665126a17cfad47" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:11.640836 9310 leader_election.cc:302] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b; no voters: 5227b26243484bbfb177a995ceb50afc, a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:11.641139 9985 raft_consensus.cc:2726] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:11.642413 6747 catalog_manager.cc:5484] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:11.647950 6747 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 07:59:11.648115 6747 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 07:59:11.651813 6747 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 20deb333e99e43bb86262ed3390d29a0
I20240503 07:59:11.652438 6747 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 0123e273efc34f538c88705829865dfd
I20240503 07:59:11.652606 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 20deb333e99e43bb86262ed3390d29a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:11.652791 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 20deb333e99e43bb86262ed3390d29a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:11.652822 9956 tablet_replica.cc:330] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:11.652820 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet 20deb333e99e43bb86262ed3390d29a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:11.652963 9959 tablet_replica.cc:330] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:11.652969 9956 raft_consensus.cc:2229] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.652992 9957 tablet_replica.cc:330] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:11.653091 9959 raft_consensus.cc:2229] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.653115 9956 raft_consensus.cc:2258] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.653115 9957 raft_consensus.cc:2229] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.653275 9957 raft_consensus.cc:2258] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.653399 9959 raft_consensus.cc:2258] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.653666 9956 ts_tablet_manager.cc:1875] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.653745 9957 ts_tablet_manager.cc:1875] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.653915 9959 ts_tablet_manager.cc:1875] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.656381 9956 ts_tablet_manager.cc:1888] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.656463 9956 log.cc:1196] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/20deb333e99e43bb86262ed3390d29a0
I20240503 07:59:11.656491 9957 ts_tablet_manager.cc:1888] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.656551 9957 log.cc:1196] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/20deb333e99e43bb86262ed3390d29a0
I20240503 07:59:11.656718 9956 ts_tablet_manager.cc:1909] T 20deb333e99e43bb86262ed3390d29a0 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:11.656761 9957 ts_tablet_manager.cc:1909] T 20deb333e99e43bb86262ed3390d29a0 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:11.657464 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 20deb333e99e43bb86262ed3390d29a0 (table rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]) successfully deleted
I20240503 07:59:11.657805 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 20deb333e99e43bb86262ed3390d29a0 (table rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]) successfully deleted
I20240503 07:59:11.658702 6747 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 7b8fccecada34616bdbf83d1f3ec9689
I20240503 07:59:11.659029 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 0123e273efc34f538c88705829865dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:11.659130 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 0123e273efc34f538c88705829865dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:11.659211 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b8fccecada34616bdbf83d1f3ec9689 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:11.659266 9958 tablet_replica.cc:330] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:11.659294 9956 tablet_replica.cc:330] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:11.659425 9956 raft_consensus.cc:2229] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.659425 9958 raft_consensus.cc:2229] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.659461 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet 0123e273efc34f538c88705829865dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:11.659560 9956 raft_consensus.cc:2258] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.659655 9957 tablet_replica.cc:330] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:11.659778 9958 raft_consensus.cc:2258] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.659911 9957 raft_consensus.cc:2229] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.659953 9959 ts_tablet_manager.cc:1888] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.660041 9959 log.cc:1196] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/20deb333e99e43bb86262ed3390d29a0
I20240503 07:59:11.660225 9956 ts_tablet_manager.cc:1875] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.660310 9959 ts_tablet_manager.cc:1909] T 20deb333e99e43bb86262ed3390d29a0 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:11.660355 9958 ts_tablet_manager.cc:1875] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.660923 9959 tablet_replica.cc:330] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:11.661059 9959 raft_consensus.cc:2229] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.661177 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 20deb333e99e43bb86262ed3390d29a0 (table rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]) successfully deleted
I20240503 07:59:11.661209 9959 raft_consensus.cc:2258] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.661671 9957 raft_consensus.cc:2258] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.661689 9959 ts_tablet_manager.cc:1875] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.662237 9957 ts_tablet_manager.cc:1875] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.662781 9958 ts_tablet_manager.cc:1888] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.662772 9956 ts_tablet_manager.cc:1888] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.662859 9956 log.cc:1196] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0123e273efc34f538c88705829865dfd
I20240503 07:59:11.662860 9958 log.cc:1196] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0123e273efc34f538c88705829865dfd
I20240503 07:59:11.663112 9958 ts_tablet_manager.cc:1909] T 0123e273efc34f538c88705829865dfd P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:11.663237 9956 ts_tablet_manager.cc:1909] T 0123e273efc34f538c88705829865dfd P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:11.663810 9959 ts_tablet_manager.cc:1888] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.664278 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 0123e273efc34f538c88705829865dfd (table rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]) successfully deleted
I20240503 07:59:11.664418 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 0123e273efc34f538c88705829865dfd (table rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]) successfully deleted
I20240503 07:59:11.664561 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b8fccecada34616bdbf83d1f3ec9689 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:11.664710 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b8fccecada34616bdbf83d1f3ec9689 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:11.664881 9956 tablet_replica.cc:330] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:11.664923 9959 log.cc:1196] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7b8fccecada34616bdbf83d1f3ec9689
I20240503 07:59:11.665051 9956 raft_consensus.cc:2229] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.665221 9956 raft_consensus.cc:2258] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.665246 9959 ts_tablet_manager.cc:1909] T 7b8fccecada34616bdbf83d1f3ec9689 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:11.665508 9957 ts_tablet_manager.cc:1888] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.665622 9957 log.cc:1196] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0123e273efc34f538c88705829865dfd
I20240503 07:59:11.665725 9956 ts_tablet_manager.cc:1875] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.665946 9957 ts_tablet_manager.cc:1909] T 0123e273efc34f538c88705829865dfd P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:11.666052 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 7b8fccecada34616bdbf83d1f3ec9689 (table rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]) successfully deleted
I20240503 07:59:11.666671 9957 tablet_replica.cc:330] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:11.666803 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 0123e273efc34f538c88705829865dfd (table rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]) successfully deleted
I20240503 07:59:11.666858 9957 raft_consensus.cc:2229] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.667202 9957 raft_consensus.cc:2258] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.667588 9956 ts_tablet_manager.cc:1888] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.667657 9956 log.cc:1196] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7b8fccecada34616bdbf83d1f3ec9689
I20240503 07:59:11.667755 9957 ts_tablet_manager.cc:1875] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.667882 9956 ts_tablet_manager.cc:1909] T 7b8fccecada34616bdbf83d1f3ec9689 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:11.668483 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 7b8fccecada34616bdbf83d1f3ec9689 (table rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]) successfully deleted
I20240503 07:59:11.669950 9957 ts_tablet_manager.cc:1888] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.670048 9957 log.cc:1196] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7b8fccecada34616bdbf83d1f3ec9689
I20240503 07:59:11.670500 9957 ts_tablet_manager.cc:1909] T 7b8fccecada34616bdbf83d1f3ec9689 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:11.671151 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 7b8fccecada34616bdbf83d1f3ec9689 (table rebalance_test_table_2 [id=235ede07fcf94c2398520c37bf8d055b]) successfully deleted
I20240503 07:59:11.698328 7279 consensus_queue.cc:237] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:11.700196 7149 raft_consensus.cc:1261] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.700551 6888 raft_consensus.cc:1261] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.700788 9053 consensus_queue.cc:1035] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.701261 9053 consensus_queue.cc:1035] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:11.704538 7215 consensus_peers.cc:485] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a00e3d35dbe5490f86bccd96a97f92f7. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.704252 9053 raft_consensus.cc:2932] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:11.704708 6888 raft_consensus.cc:2932] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:11.707908 6741 catalog_manager.cc:5484] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:11.707739 7149 raft_consensus.cc:2932] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:11.733337 6888 consensus_queue.cc:237] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:11.735242 7020 raft_consensus.cc:1261] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.735483 7149 raft_consensus.cc:1261] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.735736 10089 consensus_queue.cc:1035] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.736027 9900 consensus_queue.cc:1035] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.738075 9900 raft_consensus.cc:2932] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:11.738636 7020 raft_consensus.cc:2932] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:11.740298 7149 raft_consensus.cc:2932] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
W20240503 07:59:11.742564 6823 consensus_peers.cc:485] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f946926ddd84a2cb758a795638e3ef5. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.742484 6747 catalog_manager.cc:5484] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:11.765187 6743 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:11.766018 7279 consensus_queue.cc:237] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:11.767874 7020 raft_consensus.cc:1261] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.768074 7150 raft_consensus.cc:1261] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.768340 9053 consensus_queue.cc:1035] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.768543 9059 consensus_queue.cc:1035] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.772717 9766 raft_consensus.cc:1261] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:11.772711 9374 raft_consensus.cc:1261] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:11.773490 10109 raft_consensus.cc:2932] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:11.773912 7020 raft_consensus.cc:2932] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:11.774032 9059 consensus_queue.cc:1035] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:11.774647 7218 consensus_peers.cc:485] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd6aa70d9c4c4baea597bbb9788fdb37. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.774960 9053 consensus_queue.cc:1035] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.776372 7149 raft_consensus.cc:2932] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:11.777171 6741 catalog_manager.cc:5484] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:11.784035 6740 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet b6a74bf652c7452b9665126a17cfad47 (table rebalancer_dynamic_table [id=0261b84d392944268c2d8ef3e031ebb7]). Expected version 1 got 0
I20240503 07:59:11.785157 10121 tablet.cc:1722] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:11.785501 6747 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet b6a74bf652c7452b9665126a17cfad47 (table rebalancer_dynamic_table [id=0261b84d392944268c2d8ef3e031ebb7]). Expected version 1 got 0
I20240503 07:59:11.786190 10121 tablet.cc:1620] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:11.787307 6747 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet b6a74bf652c7452b9665126a17cfad47 (table rebalancer_dynamic_table [id=0261b84d392944268c2d8ef3e031ebb7]). Expected version 1 got 0
I20240503 07:59:11.787356 10113 mvcc.cc:205] Tried to move back new op lower bound from 7023506029654888448 to 7023506029113593856. Current Snapshot: MvccSnapshot[applied={T|T < 7023506029654888448}]
I20240503 07:59:11.788064 6740 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet b6a74bf652c7452b9665126a17cfad47 (table rebalancer_dynamic_table [id=0261b84d392944268c2d8ef3e031ebb7]). Expected version 1 got 0
I20240503 07:59:11.788100 10123 tablet.cc:1722] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:11.789258 10123 tablet.cc:1620] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:11.790028 10122 tablet.cc:1722] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:11.790963 10122 tablet.cc:1620] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:11.793269 6747 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=0261b84d392944268c2d8ef3e031ebb7] alter complete (version 1)
I20240503 07:59:11.811914 7149 consensus_queue.cc:237] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:11.813558 7020 raft_consensus.cc:1261] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.813971 7279 raft_consensus.cc:1261] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.814013 8920 consensus_queue.cc:1035] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.814429 10136 consensus_queue.cc:1035] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.816762 8920 raft_consensus.cc:2932] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:11.817268 7020 raft_consensus.cc:2932] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
W20240503 07:59:11.819820 7087 consensus_peers.cc:485] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Couldn't send request to peer a372e98a670f49e9adca46ab5c836c4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 921bd35f61ed4465aef45eee3383c9b3. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.820024 6747 catalog_manager.cc:5484] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:11.820595 7279 raft_consensus.cc:2932] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:11.841271 7149 consensus_queue.cc:237] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:11.842882 7279 raft_consensus.cc:1261] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.843199 6888 raft_consensus.cc:1261] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.843325 8920 consensus_queue.cc:1035] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.844029 8920 consensus_queue.cc:1035] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.846107 10136 raft_consensus.cc:2932] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
W20240503 07:59:11.847137 7084 consensus_peers.cc:485] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96c14f4b2a054ea98bde16bc8d4422ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.847621 6886 raft_consensus.cc:2932] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:11.848627 7279 raft_consensus.cc:2932] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:11.851577 6747 catalog_manager.cc:5484] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:11.854010 9977 consensus_queue.cc:1035] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.858397 9374 raft_consensus.cc:3037] T dae94f108f8142379c271e9fc22524ef P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.860121 10166 consensus_queue.cc:1035] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.875509 6886 consensus_queue.cc:237] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:11.877259 7149 raft_consensus.cc:1261] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:11.877383 6823 consensus_peers.cc:485] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cbcf7ab42a1463582e96927d7f16b9a. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.877487 7020 raft_consensus.cc:1261] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.877769 10089 consensus_queue.cc:1035] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.878103 9899 consensus_queue.cc:1035] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.880070 10089 raft_consensus.cc:2932] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:11.880445 7020 raft_consensus.cc:2932] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:11.882964 7150 raft_consensus.cc:2932] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:11.883551 6743 catalog_manager.cc:5484] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:11.895519 6747 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:11.901588 10121 tablet.cc:1722] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:11.901961 10121 tablet.cc:1620] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:11.902480 10122 tablet.cc:1722] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:11.902832 10122 tablet.cc:1620] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:11.903513 10123 tablet.cc:1722] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:11.903797 10123 tablet.cc:1620] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:11.910271 6747 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=0261b84d392944268c2d8ef3e031ebb7] alter complete (version 2)
I20240503 07:59:11.912444 7282 consensus_queue.cc:237] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:11.914062 7150 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.914079 7020 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:11.914233 7218 consensus_peers.cc:485] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64d33ff4164b49a7948aa5afee40bf3f. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.914518 10070 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.914690 9059 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.916923 9059 raft_consensus.cc:2932] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:11.917472 7150 raft_consensus.cc:2932] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:11.919404 7020 raft_consensus.cc:2932] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:11.920506 6747 catalog_manager.cc:5484] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:11.933804 10070 consensus_queue.cc:1035] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.937796 9059 consensus_queue.cc:1035] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.945719 6886 consensus_queue.cc:237] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:11.947352 7020 raft_consensus.cc:1261] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.947706 7282 raft_consensus.cc:1261] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.947813 10089 consensus_queue.cc:1035] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.948239 10089 consensus_queue.cc:1035] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.950083 9899 raft_consensus.cc:2932] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:11.952138 7020 raft_consensus.cc:2932] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:11.952138 7282 raft_consensus.cc:2932] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:11.952977 6743 catalog_manager.cc:5484] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
W20240503 07:59:11.954036 6825 consensus_peers.cc:485] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 -> Peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Couldn't send request to peer a372e98a670f49e9adca46ab5c836c4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eef64b6b6db54db2b47314d672269043. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.976071 9881 consensus_queue.cc:1035] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.979030 7282 consensus_queue.cc:237] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:11.980859 7020 raft_consensus.cc:1261] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.981235 9837 consensus_queue.cc:1035] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.981591 9053 consensus_queue.cc:1035] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:11.983006 7215 consensus_peers.cc:485] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a24f5c59ec4821b56163a6dd685dc9. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.983582 6886 raft_consensus.cc:1261] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.984108 9053 consensus_queue.cc:1035] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.987015 10070 raft_consensus.cc:2932] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:11.987524 6886 raft_consensus.cc:2932] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:11.991932 6740 catalog_manager.cc:5484] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:11.993178 7020 raft_consensus.cc:2932] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:12.015432 6747 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:12.021960 10121 tablet.cc:1722] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:12.022554 10123 tablet.cc:1722] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:12.025501 10122 tablet.cc:1722] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:12.026742 6886 consensus_queue.cc:237] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:12.028805 7150 raft_consensus.cc:1261] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.028808 7282 raft_consensus.cc:1261] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.029212 6747 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=0261b84d392944268c2d8ef3e031ebb7] alter complete (version 3)
I20240503 07:59:12.029278 9899 consensus_queue.cc:1035] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.029673 9899 consensus_queue.cc:1035] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:12.031180 6823 consensus_peers.cc:485] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b74fd6f87ab44bebb22167d0b6372cd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.032106 9899 raft_consensus.cc:2932] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:12.032508 7150 raft_consensus.cc:2932] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:12.034375 7282 raft_consensus.cc:2932] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:12.036249 6743 catalog_manager.cc:5484] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:12.056329 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:12.060271 7282 consensus_queue.cc:237] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.062127 6886 raft_consensus.cc:1261] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:12.062309 7218 consensus_peers.cc:485] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebc1fb490c6b448ab86bc707ee8faddc. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.062592 9053 consensus_queue.cc:1035] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.063094 7018 raft_consensus.cc:1261] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.064206 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 96289b8f77bb4241bade14b1011e1de4 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bcee57aafa9c44d5a92e47a3e072d94b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.064188 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 96289b8f77bb4241bade14b1011e1de4 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bcee57aafa9c44d5a92e47a3e072d94b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.064520 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96289b8f77bb4241bade14b1011e1de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.064523 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96289b8f77bb4241bade14b1011e1de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.064563 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 96289b8f77bb4241bade14b1011e1de4 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bcee57aafa9c44d5a92e47a3e072d94b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.064827 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96289b8f77bb4241bade14b1011e1de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.067095 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac0bb7b2c95434aa38f0f0e3718e89c (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bcee57aafa9c44d5a92e47a3e072d94b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.067276 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac0bb7b2c95434aa38f0f0e3718e89c (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bcee57aafa9c44d5a92e47a3e072d94b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.067430 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac0bb7b2c95434aa38f0f0e3718e89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.067469 10279 tablet_bootstrap.cc:492] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:12.067276 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 59a2f05167d94974ae7998b545874e06 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bcee57aafa9c44d5a92e47a3e072d94b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.067523 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac0bb7b2c95434aa38f0f0e3718e89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.067664 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a2f05167d94974ae7998b545874e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.068336 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 59a2f05167d94974ae7998b545874e06 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bcee57aafa9c44d5a92e47a3e072d94b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.068583 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a2f05167d94974ae7998b545874e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.068648 10279 tablet_bootstrap.cc:654] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.070938 10282 tablet_bootstrap.cc:492] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:12.070984 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 59a2f05167d94974ae7998b545874e06 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bcee57aafa9c44d5a92e47a3e072d94b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.071125 9059 consensus_queue.cc:1035] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.071390 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a2f05167d94974ae7998b545874e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.072001 10282 tablet_bootstrap.cc:654] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.073115 10282 tablet_bootstrap.cc:492] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:12.073215 10282 ts_tablet_manager.cc:1367] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:12.073603 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac0bb7b2c95434aa38f0f0e3718e89c (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bcee57aafa9c44d5a92e47a3e072d94b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.073654 10282 raft_consensus.cc:348] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.073765 10282 raft_consensus.cc:374] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.073794 10282 raft_consensus.cc:729] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.073881 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac0bb7b2c95434aa38f0f0e3718e89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.073912 10282 consensus_queue.cc:260] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.074201 10282 ts_tablet_manager.cc:1398] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.074182 10283 tablet_bootstrap.cc:492] T 59a2f05167d94974ae7998b545874e06 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:12.075269 10283 tablet_bootstrap.cc:654] T 59a2f05167d94974ae7998b545874e06 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.076318 10279 tablet_bootstrap.cc:492] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:12.076318 10283 tablet_bootstrap.cc:492] T 59a2f05167d94974ae7998b545874e06 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:12.076409 10283 ts_tablet_manager.cc:1367] T 59a2f05167d94974ae7998b545874e06 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:12.076411 10279 ts_tablet_manager.cc:1367] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:12.076665 10278 tablet_bootstrap.cc:492] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:12.076845 10283 raft_consensus.cc:348] T 59a2f05167d94974ae7998b545874e06 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.076845 10279 raft_consensus.cc:348] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.076952 10279 raft_consensus.cc:374] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.076952 10283 raft_consensus.cc:374] T 59a2f05167d94974ae7998b545874e06 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.076983 10279 raft_consensus.cc:729] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.076983 10283 raft_consensus.cc:729] T 59a2f05167d94974ae7998b545874e06 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.077106 10279 consensus_queue.cc:260] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.077383 10279 ts_tablet_manager.cc:1398] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.077129 10283 consensus_queue.cc:260] T 59a2f05167d94974ae7998b545874e06 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.077561 10279 tablet_bootstrap.cc:492] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:12.077633 10278 tablet_bootstrap.cc:654] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.078598 10283 ts_tablet_manager.cc:1398] T 59a2f05167d94974ae7998b545874e06 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:12.078788 10283 tablet_bootstrap.cc:492] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:12.079191 10279 tablet_bootstrap.cc:654] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.079403 10278 tablet_bootstrap.cc:492] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:12.079497 10278 ts_tablet_manager.cc:1367] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:12.080000 10278 raft_consensus.cc:348] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.080397 10280 tablet_bootstrap.cc:492] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:12.080442 10279 tablet_bootstrap.cc:492] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:12.080472 10278 raft_consensus.cc:374] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.080513 10278 raft_consensus.cc:729] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.080535 10279 ts_tablet_manager.cc:1367] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:12.080672 10278 consensus_queue.cc:260] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.080973 10278 ts_tablet_manager.cc:1398] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.081032 10279 raft_consensus.cc:348] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.081136 10279 raft_consensus.cc:374] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.081164 10279 raft_consensus.cc:729] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.081315 10279 consensus_queue.cc:260] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.081532 10278 tablet_bootstrap.cc:492] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:12.081578 10279 ts_tablet_manager.cc:1398] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:12.081418 10280 tablet_bootstrap.cc:654] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.082597 10278 tablet_bootstrap.cc:654] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.082589 10283 tablet_bootstrap.cc:654] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.083747 10283 tablet_bootstrap.cc:492] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:12.083881 10283 ts_tablet_manager.cc:1367] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:12.084244 9059 raft_consensus.cc:2932] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:12.084407 10283 raft_consensus.cc:348] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.084563 10283 raft_consensus.cc:374] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.084604 10283 raft_consensus.cc:729] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.084757 10283 consensus_queue.cc:260] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.084764 7020 raft_consensus.cc:2932] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:12.085103 10283 ts_tablet_manager.cc:1398] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.086015 10280 tablet_bootstrap.cc:492] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:12.086203 10280 ts_tablet_manager.cc:1367] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:12.087221 6886 raft_consensus.cc:2932] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:12.087394 6741 catalog_manager.cc:5484] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:12.086748 10280 raft_consensus.cc:348] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.088184 10280 raft_consensus.cc:374] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.088266 10280 raft_consensus.cc:729] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.088449 10280 consensus_queue.cc:260] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.089159 10280 ts_tablet_manager.cc:1398] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:12.089385 10280 tablet_bootstrap.cc:492] T 59a2f05167d94974ae7998b545874e06 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:12.090463 10278 tablet_bootstrap.cc:492] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:12.090507 10280 tablet_bootstrap.cc:654] T 59a2f05167d94974ae7998b545874e06 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.090642 10278 ts_tablet_manager.cc:1367] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:12.093984 10280 tablet_bootstrap.cc:492] T 59a2f05167d94974ae7998b545874e06 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:12.094085 10280 ts_tablet_manager.cc:1367] T 59a2f05167d94974ae7998b545874e06 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 07:59:12.094731 10280 raft_consensus.cc:348] T 59a2f05167d94974ae7998b545874e06 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.094868 10280 raft_consensus.cc:374] T 59a2f05167d94974ae7998b545874e06 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.094902 10280 raft_consensus.cc:729] T 59a2f05167d94974ae7998b545874e06 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.095019 10280 consensus_queue.cc:260] T 59a2f05167d94974ae7998b545874e06 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.095314 10280 ts_tablet_manager.cc:1398] T 59a2f05167d94974ae7998b545874e06 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.096833 10278 raft_consensus.cc:348] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.096956 10278 raft_consensus.cc:374] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.096987 10278 raft_consensus.cc:729] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.097160 10278 consensus_queue.cc:260] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.097661 10278 ts_tablet_manager.cc:1398] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:12.112902 7018 consensus_queue.cc:237] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.114748 6886 raft_consensus.cc:1261] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.114945 7282 raft_consensus.cc:1261] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.115427 9151 consensus_queue.cc:1035] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.115670 10310 consensus_queue.cc:1035] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.118319 9151 raft_consensus.cc:2932] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:12.118798 7282 raft_consensus.cc:2932] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:12.120982 6886 raft_consensus.cc:2932] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:12.121757 6740 catalog_manager.cc:5484] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
W20240503 07:59:12.122853 6956 consensus_peers.cc:485] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Couldn't send request to peer a372e98a670f49e9adca46ab5c836c4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85bf549243424602b6615773cf8bee33. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.130331 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:12.130519 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:12.132828 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet b6a74bf652c7452b9665126a17cfad47
I20240503 07:59:12.133664 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a74bf652c7452b9665126a17cfad47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:12.133677 9746 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a74bf652c7452b9665126a17cfad47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:12.133980 9956 tablet_replica.cc:330] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:12.134187 9956 raft_consensus.cc:2229] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.134251 10320 tablet_replica.cc:330] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:12.134274 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a74bf652c7452b9665126a17cfad47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:12.134438 9837 raft_consensus.cc:482] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:12.134543 9837 raft_consensus.cc:504] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.134665 10320 raft_consensus.cc:2229] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.134778 10321 tablet_replica.cc:330] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:12.134878 10320 raft_consensus.cc:2258] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.135026 10321 raft_consensus.cc:2229] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.135154 9837 leader_election.cc:288] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:12.135321 10321 raft_consensus.cc:2258] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.136173 9956 raft_consensus.cc:2258] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.136287 10320 ts_tablet_manager.cc:1875] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.136801 9956 ts_tablet_manager.cc:1875] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.137598 10321 ts_tablet_manager.cc:1875] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.139454 9956 ts_tablet_manager.cc:1888] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.139529 9956 log.cc:1196] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b6a74bf652c7452b9665126a17cfad47
I20240503 07:59:12.139829 9956 ts_tablet_manager.cc:1909] T b6a74bf652c7452b9665126a17cfad47 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:12.141216 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet b6a74bf652c7452b9665126a17cfad47 (table rebalancer_dynamic_table_new_name [id=0261b84d392944268c2d8ef3e031ebb7]) successfully deleted
I20240503 07:59:12.141858 10321 ts_tablet_manager.cc:1888] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.141974 10321 log.cc:1196] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/b6a74bf652c7452b9665126a17cfad47
I20240503 07:59:12.142299 10321 ts_tablet_manager.cc:1909] T b6a74bf652c7452b9665126a17cfad47 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:12.143653 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac0bb7b2c95434aa38f0f0e3718e89c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:12.143723 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet b6a74bf652c7452b9665126a17cfad47 (table rebalancer_dynamic_table_new_name [id=0261b84d392944268c2d8ef3e031ebb7]) successfully deleted
I20240503 07:59:12.143818 9374 raft_consensus.cc:2454] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:12.144141 9570 leader_election.cc:302] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:12.144748 9837 raft_consensus.cc:2781] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:12.144829 9837 raft_consensus.cc:482] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:12.144870 9837 raft_consensus.cc:3037] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.145988 9837 raft_consensus.cc:504] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.146477 9837 leader_election.cc:288] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:12.146898 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac0bb7b2c95434aa38f0f0e3718e89c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:12.147035 9374 raft_consensus.cc:3037] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.148078 9374 raft_consensus.cc:2454] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:12.148494 9570 leader_election.cc:302] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:12.148723 9837 raft_consensus.cc:2781] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:12.148805 9837 raft_consensus.cc:686] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:12.148953 9837 consensus_queue.cc:237] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.149161 10320 ts_tablet_manager.cc:1888] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.149257 10320 log.cc:1196] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/b6a74bf652c7452b9665126a17cfad47
I20240503 07:59:12.149935 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac0bb7b2c95434aa38f0f0e3718e89c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:12.149991 10320 ts_tablet_manager.cc:1909] T b6a74bf652c7452b9665126a17cfad47 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:12.150075 9766 raft_consensus.cc:2454] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:12.150300 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac0bb7b2c95434aa38f0f0e3718e89c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:12.150437 9767 raft_consensus.cc:3037] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.151358 9767 raft_consensus.cc:2454] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:12.151966 6740 catalog_manager.cc:5484] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:12.153687 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet b6a74bf652c7452b9665126a17cfad47 (table rebalancer_dynamic_table_new_name [id=0261b84d392944268c2d8ef3e031ebb7]) successfully deleted
I20240503 07:59:12.165175 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 07:59:12.165359 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 07:59:12.168263 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 7e9a8ae5e91c469e81b58a61306d922c
I20240503 07:59:12.168663 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 6d8483954eeb418db0e38021f6cb41d6
I20240503 07:59:12.168893 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9a8ae5e91c469e81b58a61306d922c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:12.168944 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet f343ec9d35a940d991f6bc279a089bae
I20240503 07:59:12.169106 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9a8ae5e91c469e81b58a61306d922c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:12.169201 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9a8ae5e91c469e81b58a61306d922c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:12.169507 10336 tablet_replica.cc:330] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:12.169487 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8483954eeb418db0e38021f6cb41d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:12.169667 9957 tablet_replica.cc:330] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:12.169756 10336 raft_consensus.cc:2229] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.169793 9957 raft_consensus.cc:2229] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.169920 9957 raft_consensus.cc:2258] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.170048 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet f343ec9d35a940d991f6bc279a089bae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:12.170102 10336 raft_consensus.cc:2258] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.170900 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8483954eeb418db0e38021f6cb41d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:12.170938 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet f343ec9d35a940d991f6bc279a089bae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:12.171098 9956 tablet_replica.cc:330] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:12.171147 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet f343ec9d35a940d991f6bc279a089bae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:12.171187 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8483954eeb418db0e38021f6cb41d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:12.171233 9956 raft_consensus.cc:2229] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.171376 9956 raft_consensus.cc:2258] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.171500 9957 ts_tablet_manager.cc:1875] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.171886 9956 ts_tablet_manager.cc:1875] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.174069 9956 ts_tablet_manager.cc:1888] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.174070 9957 ts_tablet_manager.cc:1888] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.174167 9956 log.cc:1196] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/6d8483954eeb418db0e38021f6cb41d6
I20240503 07:59:12.174167 9957 log.cc:1196] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7e9a8ae5e91c469e81b58a61306d922c
I20240503 07:59:12.174443 9957 ts_tablet_manager.cc:1909] T 7e9a8ae5e91c469e81b58a61306d922c P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:12.174445 9956 ts_tablet_manager.cc:1909] T 6d8483954eeb418db0e38021f6cb41d6 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:12.175238 9956 tablet_replica.cc:330] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:12.175403 9957 tablet_replica.cc:330] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:12.175424 9956 raft_consensus.cc:2229] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.175424 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 6d8483954eeb418db0e38021f6cb41d6 (table rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]) successfully deleted
I20240503 07:59:12.175560 9956 raft_consensus.cc:2258] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.175598 9957 raft_consensus.cc:2229] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.175813 9957 raft_consensus.cc:2258] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.176239 9956 ts_tablet_manager.cc:1875] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.176369 9957 ts_tablet_manager.cc:1875] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.176645 10335 tablet_replica.cc:330] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:12.176931 10335 raft_consensus.cc:2229] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.177102 10335 raft_consensus.cc:2258] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.177404 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 7e9a8ae5e91c469e81b58a61306d922c (table rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]) successfully deleted
I20240503 07:59:12.177735 10335 ts_tablet_manager.cc:1875] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.178884 9957 ts_tablet_manager.cc:1888] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.178992 9957 log.cc:1196] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/6d8483954eeb418db0e38021f6cb41d6
I20240503 07:59:12.179311 9957 ts_tablet_manager.cc:1909] T 6d8483954eeb418db0e38021f6cb41d6 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:12.179371 9956 ts_tablet_manager.cc:1888] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.179486 9956 log.cc:1196] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f343ec9d35a940d991f6bc279a089bae
I20240503 07:59:12.179773 9956 ts_tablet_manager.cc:1909] T f343ec9d35a940d991f6bc279a089bae P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:12.180016 9957 tablet_replica.cc:330] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:12.180119 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 6d8483954eeb418db0e38021f6cb41d6 (table rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]) successfully deleted
I20240503 07:59:12.180217 9957 raft_consensus.cc:2229] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.170765 10336 ts_tablet_manager.cc:1875] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.180385 9957 raft_consensus.cc:2258] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.180490 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet f343ec9d35a940d991f6bc279a089bae (table rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]) successfully deleted
I20240503 07:59:12.180956 9957 ts_tablet_manager.cc:1875] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.180968 10335 ts_tablet_manager.cc:1888] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.181079 10335 log.cc:1196] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7e9a8ae5e91c469e81b58a61306d922c
I20240503 07:59:12.181391 10335 ts_tablet_manager.cc:1909] T 7e9a8ae5e91c469e81b58a61306d922c P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:12.183058 10336 ts_tablet_manager.cc:1888] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.183187 10336 log.cc:1196] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7e9a8ae5e91c469e81b58a61306d922c
I20240503 07:59:12.183257 10335 tablet_replica.cc:330] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:12.183415 10335 raft_consensus.cc:2229] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.183393 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 7e9a8ae5e91c469e81b58a61306d922c (table rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]) successfully deleted
I20240503 07:59:12.183539 10336 ts_tablet_manager.cc:1909] T 7e9a8ae5e91c469e81b58a61306d922c P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:12.183686 9957 ts_tablet_manager.cc:1888] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.183727 10335 raft_consensus.cc:2258] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.183799 9957 log.cc:1196] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f343ec9d35a940d991f6bc279a089bae
I20240503 07:59:12.184070 9957 ts_tablet_manager.cc:1909] T f343ec9d35a940d991f6bc279a089bae P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:12.184188 10336 tablet_replica.cc:330] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:12.184367 10336 raft_consensus.cc:2229] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.184536 10336 raft_consensus.cc:2258] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.184639 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 7e9a8ae5e91c469e81b58a61306d922c (table rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]) successfully deleted
I20240503 07:59:12.184770 10335 ts_tablet_manager.cc:1875] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.184949 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet f343ec9d35a940d991f6bc279a089bae (table rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]) successfully deleted
I20240503 07:59:12.185117 10336 ts_tablet_manager.cc:1875] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.187078 10336 ts_tablet_manager.cc:1888] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.187129 10335 ts_tablet_manager.cc:1888] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.187165 10336 log.cc:1196] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6d8483954eeb418db0e38021f6cb41d6
I20240503 07:59:12.187187 10335 log.cc:1196] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f343ec9d35a940d991f6bc279a089bae
I20240503 07:59:12.187436 10335 ts_tablet_manager.cc:1909] T f343ec9d35a940d991f6bc279a089bae P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:12.187438 10336 ts_tablet_manager.cc:1909] T 6d8483954eeb418db0e38021f6cb41d6 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:12.188100 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 6d8483954eeb418db0e38021f6cb41d6 (table rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]) successfully deleted
I20240503 07:59:12.188153 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet f343ec9d35a940d991f6bc279a089bae (table rebalance_test_table_4 [id=8d69345503d44ab2a816b468077503a8]) successfully deleted
I20240503 07:59:12.199992 9985 raft_consensus.cc:482] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:12.200157 9985 raft_consensus.cc:504] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.200775 9985 leader_election.cc:288] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:12.209203 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289b8f77bb4241bade14b1011e1de4" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:12.209388 9505 raft_consensus.cc:2454] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:12.209734 9310 leader_election.cc:302] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:12.210302 9985 raft_consensus.cc:2781] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:12.210428 9985 raft_consensus.cc:482] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:12.210466 9985 raft_consensus.cc:3037] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.211573 9985 raft_consensus.cc:504] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.212008 9985 leader_election.cc:288] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:12.212613 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289b8f77bb4241bade14b1011e1de4" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:12.212750 9505 raft_consensus.cc:3037] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.213871 9505 raft_consensus.cc:2454] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:12.223035 9310 leader_election.cc:302] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:12.223345 9985 raft_consensus.cc:2781] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:12.223583 9985 raft_consensus.cc:686] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:12.223906 9985 consensus_queue.cc:237] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.227021 6747 catalog_manager.cc:5484] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:12.235154 6747 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:12.236784 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289b8f77bb4241bade14b1011e1de4" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:12.236856 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289b8f77bb4241bade14b1011e1de4" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:12.236963 6888 raft_consensus.cc:2454] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:12.241217 10360 ts_tablet_manager.cc:902] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:12.242278 10360 tablet_copy_client.cc:308] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:12.243468 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 37bc46947720491b9774cae8c450457c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b3da677241944a9bb4a3fa7febb7c2b3]), partition=
I20240503 07:59:12.243464 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 37bc46947720491b9774cae8c450457c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b3da677241944a9bb4a3fa7febb7c2b3]), partition=
I20240503 07:59:12.243714 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 37bc46947720491b9774cae8c450457c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b3da677241944a9bb4a3fa7febb7c2b3]), partition=
I20240503 07:59:12.243767 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37bc46947720491b9774cae8c450457c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.243774 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37bc46947720491b9774cae8c450457c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.243935 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37bc46947720491b9774cae8c450457c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.246157 10283 tablet_bootstrap.cc:492] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:12.246297 10279 tablet_bootstrap.cc:492] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:12.247313 10279 tablet_bootstrap.cc:654] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.247664 10283 tablet_bootstrap.cc:654] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.251468 10279 tablet_bootstrap.cc:492] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:12.251579 10279 ts_tablet_manager.cc:1367] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 07:59:12.252115 10279 raft_consensus.cc:348] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:12.252243 10279 raft_consensus.cc:374] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.252281 10279 raft_consensus.cc:729] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.252406 10279 consensus_queue.cc:260] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:12.252694 10279 ts_tablet_manager.cc:1398] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:12.253489 10283 tablet_bootstrap.cc:492] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:12.253598 10283 ts_tablet_manager.cc:1367] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:12.254031 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet dd6aa70d9c4c4baea597bbb9788fdb37 from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:48369)
I20240503 07:59:12.254196 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet dd6aa70d9c4c4baea597bbb9788fdb37 from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:48369: session id = cb73994515af4880b05383e78368bb4a-dd6aa70d9c4c4baea597bbb9788fdb37
I20240503 07:59:12.254169 10283 raft_consensus.cc:348] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:12.254285 10283 raft_consensus.cc:374] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.254319 10283 raft_consensus.cc:729] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.254462 10283 consensus_queue.cc:260] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:12.254865 10366 tablet_bootstrap.cc:492] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:12.255779 7302 tablet_copy_source_session.cc:205] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.256000 10366 tablet_bootstrap.cc:654] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.256657 10283 ts_tablet_manager.cc:1398] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:12.257812 10360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6aa70d9c4c4baea597bbb9788fdb37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.260319 10360 tablet_copy_client.cc:783] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.261014 10360 tablet_copy_client.cc:647] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.261619 10366 tablet_bootstrap.cc:492] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:12.261734 10366 ts_tablet_manager.cc:1367] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:12.262210 10366 raft_consensus.cc:348] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:12.262322 10366 raft_consensus.cc:374] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.262372 10366 raft_consensus.cc:729] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.262495 10366 consensus_queue.cc:260] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:12.262626 10136 raft_consensus.cc:482] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:12.262696 10136 raft_consensus.cc:504] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:12.262836 10366 ts_tablet_manager.cc:1398] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.263121 10136 leader_election.cc:288] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:12.264405 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37bc46947720491b9774cae8c450457c" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:12.264555 9767 raft_consensus.cc:2454] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:12.266207 7087 leader_election.cc:302] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:12.266712 10136 raft_consensus.cc:2781] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:12.266796 10136 raft_consensus.cc:482] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:12.266834 10136 raft_consensus.cc:3037] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.267969 10136 raft_consensus.cc:504] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:12.268376 10136 leader_election.cc:288] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:12.269780 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37bc46947720491b9774cae8c450457c" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:12.269922 9767 raft_consensus.cc:3037] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.271077 9767 raft_consensus.cc:2454] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:12.271382 10360 tablet_copy_client.cc:523] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.272610 7087 leader_election.cc:302] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:12.272811 10136 raft_consensus.cc:2781] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:12.272895 10136 raft_consensus.cc:686] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:12.273032 10136 consensus_queue.cc:237] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:12.273555 10360 tablet_bootstrap.cc:492] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:12.275655 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37bc46947720491b9774cae8c450457c" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:12.275743 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37bc46947720491b9774cae8c450457c" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:12.275811 9505 raft_consensus.cc:2454] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:12.275992 6746 catalog_manager.cc:5484] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:12.280936 10372 ts_tablet_manager.cc:902] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:12.281929 10372 tablet_copy_client.cc:308] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:12.283766 10164 raft_consensus.cc:482] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:12.283896 10164 raft_consensus.cc:504] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.285123 10164 leader_election.cc:288] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:12.288566 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 921bd35f61ed4465aef45eee3383c9b3 from peer a372e98a670f49e9adca46ab5c836c4a ({username='slave'} at 127.0.99.136:53081)
I20240503 07:59:12.288733 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 921bd35f61ed4465aef45eee3383c9b3 from peer a372e98a670f49e9adca46ab5c836c4a at {username='slave'} at 127.0.99.136:53081: session id = a372e98a670f49e9adca46ab5c836c4a-921bd35f61ed4465aef45eee3383c9b3
I20240503 07:59:12.290225 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a2f05167d94974ae7998b545874e06" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:12.290426 9767 raft_consensus.cc:2454] T 59a2f05167d94974ae7998b545874e06 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:12.290892 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a2f05167d94974ae7998b545874e06" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:12.290885 9441 leader_election.cc:302] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:12.291049 6886 raft_consensus.cc:2454] T 59a2f05167d94974ae7998b545874e06 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:12.291183 10164 raft_consensus.cc:2781] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:12.291244 10164 raft_consensus.cc:482] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:12.291277 10164 raft_consensus.cc:3037] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.292410 7171 tablet_copy_source_session.cc:205] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.292435 10164 raft_consensus.cc:504] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.292800 10164 leader_election.cc:288] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:12.293296 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a2f05167d94974ae7998b545874e06" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:12.293438 9767 raft_consensus.cc:3037] T 59a2f05167d94974ae7998b545874e06 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.293412 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a2f05167d94974ae7998b545874e06" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:12.293538 6886 raft_consensus.cc:3037] T 59a2f05167d94974ae7998b545874e06 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.294550 9767 raft_consensus.cc:2454] T 59a2f05167d94974ae7998b545874e06 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:12.294726 6886 raft_consensus.cc:2454] T 59a2f05167d94974ae7998b545874e06 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:12.294746 10372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921bd35f61ed4465aef45eee3383c9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.295249 9438 leader_election.cc:302] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:12.295665 10164 raft_consensus.cc:2781] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:12.295869 10164 raft_consensus.cc:686] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:12.296119 10164 consensus_queue.cc:237] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.297223 10372 tablet_copy_client.cc:783] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.297564 10372 tablet_copy_client.cc:647] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.298942 6746 catalog_manager.cc:5484] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:12.299675 10372 tablet_copy_client.cc:523] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.301857 10372 tablet_bootstrap.cc:492] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:12.302150 10381 ts_tablet_manager.cc:902] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:12.302855 10381 tablet_copy_client.cc:308] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:12.303400 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet 64d33ff4164b49a7948aa5afee40bf3f from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:48369)
I20240503 07:59:12.303509 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet 64d33ff4164b49a7948aa5afee40bf3f from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:48369: session id = cb73994515af4880b05383e78368bb4a-64d33ff4164b49a7948aa5afee40bf3f
I20240503 07:59:12.304566 7302 tablet_copy_source_session.cc:205] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.305586 10381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d33ff4164b49a7948aa5afee40bf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.309530 10381 tablet_copy_client.cc:783] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.309779 10381 tablet_copy_client.cc:647] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.313494 10381 tablet_copy_client.cc:523] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.315340 10360 tablet_bootstrap.cc:492] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.315811 10360 tablet_bootstrap.cc:492] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:12.316097 10360 ts_tablet_manager.cc:1367] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.043s user 0.026s sys 0.006s
I20240503 07:59:12.316651 10360 raft_consensus.cc:348] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.316773 10360 raft_consensus.cc:729] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.316865 10381 tablet_bootstrap.cc:492] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:12.316932 10360 consensus_queue.cc:260] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.317979 10360 ts_tablet_manager.cc:1398] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:12.318691 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-dd6aa70d9c4c4baea597bbb9788fdb37 received from {username='slave'} at 127.0.99.134:48369
I20240503 07:59:12.318812 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session cb73994515af4880b05383e78368bb4a-dd6aa70d9c4c4baea597bbb9788fdb37 on tablet dd6aa70d9c4c4baea597bbb9788fdb37 with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:12.326579 10399 ts_tablet_manager.cc:902] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:12.328066 10399 tablet_copy_client.cc:308] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:12.331856 7129 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:12.333532 10372 tablet_bootstrap.cc:492] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.334146 10372 tablet_bootstrap.cc:492] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap complete.
I20240503 07:59:12.334424 10372 ts_tablet_manager.cc:1367] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.033s user 0.030s sys 0.001s
I20240503 07:59:12.335125 10372 raft_consensus.cc:348] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.335302 10372 raft_consensus.cc:729] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.335469 10372 consensus_queue.cc:260] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.335806 10372 ts_tablet_manager.cc:1398] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:12.336692 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session a372e98a670f49e9adca46ab5c836c4a-921bd35f61ed4465aef45eee3383c9b3 received from {username='slave'} at 127.0.99.136:53081
I20240503 07:59:12.336808 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session a372e98a670f49e9adca46ab5c836c4a-921bd35f61ed4465aef45eee3383c9b3 on tablet 921bd35f61ed4465aef45eee3383c9b3 with peer a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:12.337220 9350 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:12.342794 9616 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:12.344399 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet a00e3d35dbe5490f86bccd96a97f92f7 from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:43259)
I20240503 07:59:12.344522 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet a00e3d35dbe5490f86bccd96a97f92f7 from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:43259: session id = 159177b3caeb434baa98e04136d050b2-a00e3d35dbe5490f86bccd96a97f92f7
I20240503 07:59:12.345547 7302 tablet_copy_source_session.cc:205] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.360370 6998 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:12.360746 10399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00e3d35dbe5490f86bccd96a97f92f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.361374 10381 tablet_bootstrap.cc:492] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.361774 10381 tablet_bootstrap.cc:492] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:12.362030 10381 ts_tablet_manager.cc:1367] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.046s user 0.022s sys 0.004s
I20240503 07:59:12.362586 10381 raft_consensus.cc:348] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.362746 10381 raft_consensus.cc:729] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.362921 10381 consensus_queue.cc:260] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.363235 10381 ts_tablet_manager.cc:1398] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.363745 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-64d33ff4164b49a7948aa5afee40bf3f received from {username='slave'} at 127.0.99.134:48369
I20240503 07:59:12.363827 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session cb73994515af4880b05383e78368bb4a-64d33ff4164b49a7948aa5afee40bf3f on tablet 64d33ff4164b49a7948aa5afee40bf3f with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:12.364607 10405 ts_tablet_manager.cc:902] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:12.365537 10405 tablet_copy_client.cc:308] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:12.366453 6909 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet 7f946926ddd84a2cb758a795638e3ef5 from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:38021)
I20240503 07:59:12.366598 6909 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet 7f946926ddd84a2cb758a795638e3ef5 from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:38021: session id = 3befbb2d206f4be89d3e07f38837374b-7f946926ddd84a2cb758a795638e3ef5
I20240503 07:59:12.368049 6909 tablet_copy_source_session.cc:205] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.369112 9746 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:12.370319 7261 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:12.382740 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:12.383293 10405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f946926ddd84a2cb758a795638e3ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.387100 10399 tablet_copy_client.cc:783] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.387359 10399 tablet_copy_client.cc:647] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.387907 9485 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:12.389161 10399 tablet_copy_client.cc:523] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.391641 10399 tablet_bootstrap.cc:492] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:12.401998 10405 tablet_copy_client.cc:783] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.402454 10405 tablet_copy_client.cc:647] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.405140 9767 raft_consensus.cc:1261] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:12.405721 10136 consensus_queue.cc:1035] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.406626 9504 raft_consensus.cc:3037] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.407920 9504 raft_consensus.cc:1261] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:12.409693 10136 consensus_queue.cc:1035] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.410389 10119 mvcc.cc:205] Tried to move back new op lower bound from 7023506032245846016 to 7023506031712022528. Current Snapshot: MvccSnapshot[applied={T|T < 7023506032245846016}]
I20240503 07:59:12.410612 10122 tablet.cc:1722] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:12.410935 10122 tablet.cc:1620] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:12.410871 10413 tablet.cc:1722] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:12.413218 10167 mvcc.cc:205] Tried to move back new op lower bound from 7023506032245846016 to 7023506031712022528. Current Snapshot: MvccSnapshot[applied={T|T < 7023506032245846016}]
I20240503 07:59:12.413703 6746 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 37bc46947720491b9774cae8c450457c (table rebalancer_dynamic_table [id=b3da677241944a9bb4a3fa7febb7c2b3]). Expected version 1 got 0
I20240503 07:59:12.417675 6746 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 37bc46947720491b9774cae8c450457c (table rebalancer_dynamic_table [id=b3da677241944a9bb4a3fa7febb7c2b3]). Expected version 1 got 0
I20240503 07:59:12.426221 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=b3da677241944a9bb4a3fa7febb7c2b3] alter complete (version 1)
I20240503 07:59:12.427698 10405 tablet_copy_client.cc:523] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.429869 10405 tablet_bootstrap.cc:492] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:12.439544 10413 tablet.cc:1620] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:12.441468 6868 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:12.444607 10416 ts_tablet_manager.cc:902] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:12.445237 10416 tablet_copy_client.cc:308] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:12.449347 10415 tablet.cc:1722] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:12.449687 6909 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet b74fd6f87ab44bebb22167d0b6372cd1 from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:38021)
I20240503 07:59:12.449831 6909 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet b74fd6f87ab44bebb22167d0b6372cd1 from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:38021: session id = 3befbb2d206f4be89d3e07f38837374b-b74fd6f87ab44bebb22167d0b6372cd1
I20240503 07:59:12.451246 6909 tablet_copy_source_session.cc:205] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.451560 10415 tablet.cc:1620] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:12.454989 10416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74fd6f87ab44bebb22167d0b6372cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.457448 10416 tablet_copy_client.cc:783] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.457635 10416 tablet_copy_client.cc:647] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.465574 10372 ts_tablet_manager.cc:902] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:12.466136 10422 ts_tablet_manager.cc:902] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:12.466192 10372 tablet_copy_client.cc:308] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:12.467121 10422 tablet_copy_client.cc:308] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:12.467795 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet 77a24f5c59ec4821b56163a6dd685dc9 from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:43259)
I20240503 07:59:12.467929 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet 77a24f5c59ec4821b56163a6dd685dc9 from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:43259: session id = 159177b3caeb434baa98e04136d050b2-77a24f5c59ec4821b56163a6dd685dc9
I20240503 07:59:12.469003 7302 tablet_copy_source_session.cc:205] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.470227 10422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a24f5c59ec4821b56163a6dd685dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.472256 10423 ts_tablet_manager.cc:902] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:12.473368 10422 tablet_copy_client.cc:783] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.473609 10422 tablet_copy_client.cc:647] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.474198 10423 tablet_copy_client.cc:308] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:12.475234 10416 tablet_copy_client.cc:523] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.475641 10422 tablet_copy_client.cc:523] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.476894 10416 tablet_bootstrap.cc:492] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:12.477113 10422 tablet_bootstrap.cc:492] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:12.477183 10405 tablet_bootstrap.cc:492] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.477510 10405 tablet_bootstrap.cc:492] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:12.477681 10405 ts_tablet_manager.cc:1367] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.048s user 0.027s sys 0.005s
I20240503 07:59:12.478205 10405 raft_consensus.cc:348] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:12.478335 10405 raft_consensus.cc:729] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.478549 10405 consensus_queue.cc:260] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:12.479107 6909 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet 5cbcf7ab42a1463582e96927d7f16b9a from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:38021)
I20240503 07:59:12.479266 6909 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet 5cbcf7ab42a1463582e96927d7f16b9a from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:38021: session id = 3befbb2d206f4be89d3e07f38837374b-5cbcf7ab42a1463582e96927d7f16b9a
I20240503 07:59:12.479375 10405 ts_tablet_manager.cc:1398] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:12.480360 10424 ts_tablet_manager.cc:902] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:12.480360 6909 tablet_copy_source_session.cc:205] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.480947 10424 tablet_copy_client.cc:308] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:12.483487 10399 tablet_bootstrap.cc:492] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.483847 10399 tablet_bootstrap.cc:492] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:12.484033 10399 ts_tablet_manager.cc:1367] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.092s user 0.032s sys 0.000s
I20240503 07:59:12.484516 10399 raft_consensus.cc:348] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.484634 10399 raft_consensus.cc:729] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.484766 10399 consensus_queue.cc:260] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.485035 10399 ts_tablet_manager.cc:1398] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:12.485689 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-a00e3d35dbe5490f86bccd96a97f92f7 received from {username='slave'} at 127.0.99.135:43259
I20240503 07:59:12.485806 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session 159177b3caeb434baa98e04136d050b2-a00e3d35dbe5490f86bccd96a97f92f7 on tablet a00e3d35dbe5490f86bccd96a97f92f7 with peer 159177b3caeb434baa98e04136d050b2
I20240503 07:59:12.486450 6908 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet eef64b6b6db54db2b47314d672269043 from peer a372e98a670f49e9adca46ab5c836c4a ({username='slave'} at 127.0.99.136:39953)
I20240503 07:59:12.486531 6909 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-7f946926ddd84a2cb758a795638e3ef5 received from {username='slave'} at 127.0.99.133:38021
I20240503 07:59:12.486622 6909 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-7f946926ddd84a2cb758a795638e3ef5 on tablet 7f946926ddd84a2cb758a795638e3ef5 with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:12.487475 6908 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet eef64b6b6db54db2b47314d672269043 from peer a372e98a670f49e9adca46ab5c836c4a at {username='slave'} at 127.0.99.136:39953: session id = a372e98a670f49e9adca46ab5c836c4a-eef64b6b6db54db2b47314d672269043
I20240503 07:59:12.487684 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 96c14f4b2a054ea98bde16bc8d4422ac from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:60639)
I20240503 07:59:12.487798 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 96c14f4b2a054ea98bde16bc8d4422ac from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:60639: session id = 159177b3caeb434baa98e04136d050b2-96c14f4b2a054ea98bde16bc8d4422ac
I20240503 07:59:12.488569 10423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbcf7ab42a1463582e96927d7f16b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.488617 6908 tablet_copy_source_session.cc:205] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.488787 7171 tablet_copy_source_session.cc:205] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.489809 10424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c14f4b2a054ea98bde16bc8d4422ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.491541 10423 tablet_copy_client.cc:783] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.491777 10423 tablet_copy_client.cc:647] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.492446 10424 tablet_copy_client.cc:783] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.492554 10372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef64b6b6db54db2b47314d672269043. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.492642 10424 tablet_copy_client.cc:647] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.493975 10424 tablet_copy_client.cc:523] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.495894 10424 tablet_bootstrap.cc:492] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:12.497331 10423 tablet_copy_client.cc:523] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.499040 10372 tablet_copy_client.cc:783] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.499331 10372 tablet_copy_client.cc:647] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.500972 10372 tablet_copy_client.cc:523] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.502605 10372 tablet_bootstrap.cc:492] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:12.506116 10423 tablet_bootstrap.cc:492] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:12.525947 10422 tablet_bootstrap.cc:492] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.526628 10422 tablet_bootstrap.cc:492] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:12.526968 10422 ts_tablet_manager.cc:1367] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.050s user 0.027s sys 0.001s
I20240503 07:59:12.529673 10422 raft_consensus.cc:348] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.530763 10372 tablet_bootstrap.cc:492] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.529935 10422 raft_consensus.cc:729] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.531540 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:12.531617 10422 consensus_queue.cc:260] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.533186 10422 ts_tablet_manager.cc:1398] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240503 07:59:12.534385 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-77a24f5c59ec4821b56163a6dd685dc9 received from {username='slave'} at 127.0.99.135:43259
I20240503 07:59:12.534489 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session 159177b3caeb434baa98e04136d050b2-77a24f5c59ec4821b56163a6dd685dc9 on tablet 77a24f5c59ec4821b56163a6dd685dc9 with peer 159177b3caeb434baa98e04136d050b2
I20240503 07:59:12.537091 10372 tablet_bootstrap.cc:492] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap complete.
I20240503 07:59:12.537395 10372 ts_tablet_manager.cc:1367] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.035s user 0.020s sys 0.007s
I20240503 07:59:12.537982 10372 raft_consensus.cc:348] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.538208 10372 raft_consensus.cc:729] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.538443 10372 consensus_queue.cc:260] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.538925 10372 ts_tablet_manager.cc:1398] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:12.540167 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session a372e98a670f49e9adca46ab5c836c4a-eef64b6b6db54db2b47314d672269043 received from {username='slave'} at 127.0.99.136:39953
I20240503 07:59:12.540268 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session a372e98a670f49e9adca46ab5c836c4a-eef64b6b6db54db2b47314d672269043 on tablet eef64b6b6db54db2b47314d672269043 with peer a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:12.541375 10413 tablet.cc:1722] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:12.541752 10413 tablet.cc:1620] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:12.544203 10415 tablet.cc:1722] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:12.544540 10415 tablet.cc:1620] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:12.545852 10122 tablet.cc:1722] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:12.546168 10122 tablet.cc:1620] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:12.546727 10424 tablet_bootstrap.cc:492] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.547091 10424 tablet_bootstrap.cc:492] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:12.547305 10424 ts_tablet_manager.cc:1367] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.051s user 0.025s sys 0.003s
I20240503 07:59:12.547814 10424 raft_consensus.cc:348] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.547969 10424 raft_consensus.cc:729] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.548127 10424 consensus_queue.cc:260] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.548619 10424 ts_tablet_manager.cc:1398] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.549513 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-96c14f4b2a054ea98bde16bc8d4422ac received from {username='slave'} at 127.0.99.135:60639
I20240503 07:59:12.549602 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session 159177b3caeb434baa98e04136d050b2-96c14f4b2a054ea98bde16bc8d4422ac on tablet 96c14f4b2a054ea98bde16bc8d4422ac with peer 159177b3caeb434baa98e04136d050b2
I20240503 07:59:12.552240 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=b3da677241944a9bb4a3fa7febb7c2b3] alter complete (version 2)
I20240503 07:59:12.553925 10416 tablet_bootstrap.cc:492] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.554276 10416 tablet_bootstrap.cc:492] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:12.554482 10416 ts_tablet_manager.cc:1367] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.078s user 0.027s sys 0.000s
I20240503 07:59:12.554921 10416 raft_consensus.cc:348] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:12.555035 10416 raft_consensus.cc:729] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.555189 10416 consensus_queue.cc:260] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:12.555490 10416 ts_tablet_manager.cc:1398] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:12.556465 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-b74fd6f87ab44bebb22167d0b6372cd1 received from {username='slave'} at 127.0.99.133:38021
I20240503 07:59:12.556576 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-b74fd6f87ab44bebb22167d0b6372cd1 on tablet b74fd6f87ab44bebb22167d0b6372cd1 with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:12.562361 10423 tablet_bootstrap.cc:492] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.562695 10423 tablet_bootstrap.cc:492] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:12.562906 10423 ts_tablet_manager.cc:1367] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.057s user 0.020s sys 0.005s
I20240503 07:59:12.563412 10423 raft_consensus.cc:348] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:12.563560 10423 raft_consensus.cc:729] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.563760 10423 consensus_queue.cc:260] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:12.564185 10423 ts_tablet_manager.cc:1398] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.564918 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-5cbcf7ab42a1463582e96927d7f16b9a received from {username='slave'} at 127.0.99.133:38021
I20240503 07:59:12.565019 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-5cbcf7ab42a1463582e96927d7f16b9a on tablet 5cbcf7ab42a1463582e96927d7f16b9a with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:12.568940 10381 ts_tablet_manager.cc:902] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:12.569535 10381 tablet_copy_client.cc:308] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:12.570029 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet ebc1fb490c6b448ab86bc707ee8faddc from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:48369)
I20240503 07:59:12.570138 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet ebc1fb490c6b448ab86bc707ee8faddc from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:48369: session id = cb73994515af4880b05383e78368bb4a-ebc1fb490c6b448ab86bc707ee8faddc
I20240503 07:59:12.571049 7302 tablet_copy_source_session.cc:205] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.572197 10381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc1fb490c6b448ab86bc707ee8faddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.574398 10381 tablet_copy_client.cc:783] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.574699 10381 tablet_copy_client.cc:647] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.576112 10381 tablet_copy_client.cc:523] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.577658 10381 tablet_bootstrap.cc:492] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:12.579548 7018 consensus_queue.cc:237] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.581460 7149 raft_consensus.cc:1261] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.582453 10310 consensus_queue.cc:1035] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:12.585217 6953 consensus_peers.cc:485] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b424a8b7bba44b098731f8ee18e8c9b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.585372 6886 raft_consensus.cc:1261] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.586102 10310 consensus_queue.cc:1035] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.588012 9151 raft_consensus.cc:2932] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:12.590106 7150 raft_consensus.cc:2932] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:12.592044 6740 catalog_manager.cc:5484] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:12.594627 6886 raft_consensus.cc:2932] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:12.606285 10381 tablet_bootstrap.cc:492] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.606693 10381 tablet_bootstrap.cc:492] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:12.606949 10381 ts_tablet_manager.cc:1367] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.001s
I20240503 07:59:12.607489 10381 raft_consensus.cc:348] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.607714 10381 raft_consensus.cc:729] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.607940 10381 consensus_queue.cc:260] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.608394 10381 ts_tablet_manager.cc:1398] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.608992 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-ebc1fb490c6b448ab86bc707ee8faddc received from {username='slave'} at 127.0.99.134:48369
I20240503 07:59:12.609082 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session cb73994515af4880b05383e78368bb4a-ebc1fb490c6b448ab86bc707ee8faddc on tablet ebc1fb490c6b448ab86bc707ee8faddc with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:12.623590 7282 consensus_queue.cc:237] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:12.625663 7150 raft_consensus.cc:1261] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:12.625888 7216 consensus_peers.cc:485] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b916e89edaf046b99767e5cca531bb8b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.626056 6886 raft_consensus.cc:1261] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.626271 9053 consensus_queue.cc:1035] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.626739 9053 consensus_queue.cc:1035] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.629209 10070 raft_consensus.cc:2932] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:12.629714 6886 raft_consensus.cc:2932] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:12.633358 6744 catalog_manager.cc:5484] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:12.633422 7150 raft_consensus.cc:2932] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:12.655047 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:12.663282 10413 tablet.cc:1722] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:12.663789 10415 tablet.cc:1722] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:12.664520 10122 tablet.cc:1722] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:12.665480 7018 consensus_queue.cc:237] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:12.668457 7278 raft_consensus.cc:1261] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.669076 7149 raft_consensus.cc:1261] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.669909 10469 consensus_queue.cc:1035] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.670258 10310 consensus_queue.cc:1035] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.671021 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=b3da677241944a9bb4a3fa7febb7c2b3] alter complete (version 3)
W20240503 07:59:12.673447 6956 consensus_peers.cc:485] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82fc9f4263244c6fa2fb39543230eb87. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.675834 9767 raft_consensus.cc:1203] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.676707 10310 raft_consensus.cc:2932] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:12.677819 7278 raft_consensus.cc:2932] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:12.678746 6746 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 07:59:12.678885 6746 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 07:59:12.679642 7149 raft_consensus.cc:2932] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:12.681991 6746 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 4 replicas of tablet 77a24f5c59ec4821b56163a6dd685dc9
I20240503 07:59:12.682087 6740 catalog_manager.cc:5484] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:12.682660 6746 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 0d2c2c5aafd640ccac547de40ec34841
I20240503 07:59:12.682880 6746 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet a8db24c1bdb142c2b2a6b3041f308892
I20240503 07:59:12.683250 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a24f5c59ec4821b56163a6dd685dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:12.683413 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a24f5c59ec4821b56163a6dd685dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:12.683552 10336 tablet_replica.cc:330] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:12.683583 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2c2c5aafd640ccac547de40ec34841 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:12.683718 10336 raft_consensus.cc:2229] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.683725 9616 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a24f5c59ec4821b56163a6dd685dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:12.683940 10336 raft_consensus.cc:2258] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.684041 9957 tablet_replica.cc:330] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:12.684077 10511 tablet_replica.cc:330] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:12.684190 10511 raft_consensus.cc:2229] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.684190 9957 raft_consensus.cc:2229] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.684288 10511 raft_consensus.cc:2258] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.684358 9957 raft_consensus.cc:2258] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.684428 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet a8db24c1bdb142c2b2a6b3041f308892 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:12.684561 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2c2c5aafd640ccac547de40ec34841 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:12.684733 10335 tablet_replica.cc:330] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:12.684828 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet a8db24c1bdb142c2b2a6b3041f308892 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:12.684867 10335 raft_consensus.cc:2229] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.685097 10335 raft_consensus.cc:2258] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.685217 10336 ts_tablet_manager.cc:1875] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.685379 9957 ts_tablet_manager.cc:1875] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.685595 10335 ts_tablet_manager.cc:1875] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.687423 10336 ts_tablet_manager.cc:1888] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.687492 10336 log.cc:1196] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/77a24f5c59ec4821b56163a6dd685dc9
I20240503 07:59:12.687722 10336 ts_tablet_manager.cc:1909] T 77a24f5c59ec4821b56163a6dd685dc9 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:12.687826 9957 ts_tablet_manager.cc:1888] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.687826 10335 ts_tablet_manager.cc:1888] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.687899 9957 log.cc:1196] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/77a24f5c59ec4821b56163a6dd685dc9
I20240503 07:59:12.687899 10335 log.cc:1196] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0d2c2c5aafd640ccac547de40ec34841
I20240503 07:59:12.688144 10335 ts_tablet_manager.cc:1909] T 0d2c2c5aafd640ccac547de40ec34841 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:12.688153 9957 ts_tablet_manager.cc:1909] T 77a24f5c59ec4821b56163a6dd685dc9 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:12.688419 10336 tablet_replica.cc:330] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:12.688640 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a24f5c59ec4821b56163a6dd685dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:12.688735 7262 tablet_service.cc:1514] Processing DeleteTablet for tablet a8db24c1bdb142c2b2a6b3041f308892 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:12.688735 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2c2c5aafd640ccac547de40ec34841 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:12.690598 10511 ts_tablet_manager.cc:1875] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.690690 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 77a24f5c59ec4821b56163a6dd685dc9 (table rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]) successfully deleted
I20240503 07:59:12.690896 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 0d2c2c5aafd640ccac547de40ec34841 (table rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]) successfully deleted
I20240503 07:59:12.691215 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 77a24f5c59ec4821b56163a6dd685dc9 (table rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]) successfully deleted
I20240503 07:59:12.691407 10336 raft_consensus.cc:2229] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.691627 9957 tablet_replica.cc:330] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:12.692946 10512 tablet_replica.cc:330] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:12.693071 9957 raft_consensus.cc:2229] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.693204 10336 raft_consensus.cc:2258] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.693405 9957 raft_consensus.cc:2258] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.693665 10511 ts_tablet_manager.cc:1888] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.694061 10511 log.cc:1196] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/77a24f5c59ec4821b56163a6dd685dc9
I20240503 07:59:12.694494 10511 ts_tablet_manager.cc:1909] T 77a24f5c59ec4821b56163a6dd685dc9 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:12.694494 10512 raft_consensus.cc:2229] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.694809 6886 raft_consensus.cc:3037] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.695462 9957 ts_tablet_manager.cc:1875] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.695776 10164 consensus_queue.cc:1035] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.696660 9985 consensus_queue.cc:1035] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.696800 10512 raft_consensus.cc:2258] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.697391 10336 ts_tablet_manager.cc:1875] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.698257 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 77a24f5c59ec4821b56163a6dd685dc9 (table rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]) successfully deleted
I20240503 07:59:12.700944 10512 ts_tablet_manager.cc:1875] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.701375 10336 ts_tablet_manager.cc:1888] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.701509 9957 ts_tablet_manager.cc:1888] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.701654 9957 log.cc:1196] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0d2c2c5aafd640ccac547de40ec34841
I20240503 07:59:12.701779 10336 log.cc:1196] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a8db24c1bdb142c2b2a6b3041f308892
I20240503 07:59:12.701786 9985 consensus_queue.cc:1035] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.702112 9957 ts_tablet_manager.cc:1909] T 0d2c2c5aafd640ccac547de40ec34841 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:12.702790 9957 tablet_replica.cc:330] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:12.703116 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 0d2c2c5aafd640ccac547de40ec34841 (table rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]) successfully deleted
I20240503 07:59:12.703842 10512 ts_tablet_manager.cc:1888] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.703979 10512 log.cc:1196] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/77a24f5c59ec4821b56163a6dd685dc9
I20240503 07:59:12.704268 10512 ts_tablet_manager.cc:1909] T 77a24f5c59ec4821b56163a6dd685dc9 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:12.704862 10336 ts_tablet_manager.cc:1909] T a8db24c1bdb142c2b2a6b3041f308892 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:12.705544 10512 tablet_replica.cc:330] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:12.705570 9957 raft_consensus.cc:2229] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.705719 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet a8db24c1bdb142c2b2a6b3041f308892 (table rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]) successfully deleted
I20240503 07:59:12.705772 9957 raft_consensus.cc:2258] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.705798 10512 raft_consensus.cc:2229] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.705942 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 77a24f5c59ec4821b56163a6dd685dc9 (table rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]) successfully deleted
I20240503 07:59:12.706329 9957 ts_tablet_manager.cc:1875] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.706604 10512 raft_consensus.cc:2258] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.707171 10512 ts_tablet_manager.cc:1875] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.708302 9957 ts_tablet_manager.cc:1888] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.708376 9957 log.cc:1196] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a8db24c1bdb142c2b2a6b3041f308892
I20240503 07:59:12.708636 9957 ts_tablet_manager.cc:1909] T a8db24c1bdb142c2b2a6b3041f308892 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:12.708904 10512 ts_tablet_manager.cc:1888] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.708973 10512 log.cc:1196] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a8db24c1bdb142c2b2a6b3041f308892
I20240503 07:59:12.709208 10512 ts_tablet_manager.cc:1909] T a8db24c1bdb142c2b2a6b3041f308892 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:12.709797 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet a8db24c1bdb142c2b2a6b3041f308892 (table rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]) successfully deleted
I20240503 07:59:12.711755 10164 consensus_queue.cc:1035] T 59a2f05167d94974ae7998b545874e06 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.712621 10512 tablet_replica.cc:330] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:12.712931 10512 raft_consensus.cc:2229] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.713150 10512 raft_consensus.cc:2258] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.713748 10512 ts_tablet_manager.cc:1875] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.713829 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet a8db24c1bdb142c2b2a6b3041f308892 (table rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]) successfully deleted
I20240503 07:59:12.716346 10512 ts_tablet_manager.cc:1888] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.716500 10512 log.cc:1196] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0d2c2c5aafd640ccac547de40ec34841
I20240503 07:59:12.716820 10512 ts_tablet_manager.cc:1909] T 0d2c2c5aafd640ccac547de40ec34841 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:12.717954 9881 consensus_queue.cc:1035] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.719591 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 0d2c2c5aafd640ccac547de40ec34841 (table rebalance_test_table_6 [id=a8cb727c0a5d4461b9fac2e2af74cba9]) successfully deleted
I20240503 07:59:12.722762 9837 consensus_queue.cc:1035] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.724294 10372 ts_tablet_manager.cc:902] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: Initiating tablet copy from peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:12.724846 10372 tablet_copy_client.cc:308] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46383
I20240503 07:59:12.729494 7040 tablet_copy_service.cc:140] P 9c085a3241f14cb7ba1332b508f4b57d: Received BeginTabletCopySession request for tablet 85bf549243424602b6615773cf8bee33 from peer a372e98a670f49e9adca46ab5c836c4a ({username='slave'} at 127.0.99.136:40851)
I20240503 07:59:12.729650 7040 tablet_copy_service.cc:161] P 9c085a3241f14cb7ba1332b508f4b57d: Beginning new tablet copy session on tablet 85bf549243424602b6615773cf8bee33 from peer a372e98a670f49e9adca46ab5c836c4a at {username='slave'} at 127.0.99.136:40851: session id = a372e98a670f49e9adca46ab5c836c4a-85bf549243424602b6615773cf8bee33
I20240503 07:59:12.731154 7040 tablet_copy_source_session.cc:205] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.732645 10372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bf549243424602b6615773cf8bee33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.735474 10372 tablet_copy_client.cc:783] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.735690 10372 tablet_copy_client.cc:647] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.737645 10372 tablet_copy_client.cc:523] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.739289 10372 tablet_bootstrap.cc:492] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:12.741860 7149 consensus_queue.cc:237] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.743674 7282 raft_consensus.cc:1261] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.743809 7018 raft_consensus.cc:1261] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:12.744100 7087 consensus_peers.cc:485] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Couldn't send request to peer a372e98a670f49e9adca46ab5c836c4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e8e04c50d7147af903d7113df3d9744. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.744370 10505 consensus_queue.cc:1035] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.744762 10505 consensus_queue.cc:1035] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.746627 10507 raft_consensus.cc:2932] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:12.747066 7282 raft_consensus.cc:2932] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:12.749529 7018 raft_consensus.cc:2932] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:12.749912 6746 catalog_manager.cc:5484] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:12.762279 9504 raft_consensus.cc:1203] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.762846 10372 tablet_bootstrap.cc:492] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.763137 10372 tablet_bootstrap.cc:492] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap complete.
I20240503 07:59:12.763348 10372 ts_tablet_manager.cc:1367] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.004s
I20240503 07:59:12.763859 10372 raft_consensus.cc:348] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.763984 10372 raft_consensus.cc:729] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.764125 10372 consensus_queue.cc:260] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.764446 10372 ts_tablet_manager.cc:1398] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:12.765926 7040 tablet_copy_service.cc:342] P 9c085a3241f14cb7ba1332b508f4b57d: Request end of tablet copy session a372e98a670f49e9adca46ab5c836c4a-85bf549243424602b6615773cf8bee33 received from {username='slave'} at 127.0.99.136:40851
I20240503 07:59:12.766049 7040 tablet_copy_service.cc:433] P 9c085a3241f14cb7ba1332b508f4b57d: ending tablet copy session a372e98a670f49e9adca46ab5c836c4a-85bf549243424602b6615773cf8bee33 on tablet 85bf549243424602b6615773cf8bee33 with peer a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:12.773471 9504 raft_consensus.cc:1203] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.777539 7018 consensus_queue.cc:237] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.779346 7150 raft_consensus.cc:1261] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.779426 7282 raft_consensus.cc:1261] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:12.779721 6953 consensus_peers.cc:485] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31c76213f3ec451084ae7b0551cd7b48. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.779771 10469 consensus_queue.cc:1035] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.780011 9090 consensus_queue.cc:1035] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.782119 9090 raft_consensus.cc:2932] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:12.782580 6744 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:12.782524 7150 raft_consensus.cc:2932] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:12.782737 6744 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:12.785156 6744 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 37bc46947720491b9774cae8c450457c
I20240503 07:59:12.785038 7282 raft_consensus.cc:2932] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:12.785681 6746 catalog_manager.cc:5484] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:12.787220 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet 37bc46947720491b9774cae8c450457c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:12.787525 9746 tablet_service.cc:1514] Processing DeleteTablet for tablet 37bc46947720491b9774cae8c450457c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:12.788329 10565 tablet_replica.cc:330] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:12.788538 10565 raft_consensus.cc:2229] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.788744 10565 raft_consensus.cc:2258] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.789892 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 37bc46947720491b9774cae8c450457c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:12.790153 10565 ts_tablet_manager.cc:1875] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.790287 10564 tablet_replica.cc:330] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:12.790446 10564 raft_consensus.cc:2229] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.790526 10335 tablet_replica.cc:330] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:12.790614 10564 raft_consensus.cc:2258] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.790692 10335 raft_consensus.cc:2229] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.790920 10335 raft_consensus.cc:2258] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.791219 10564 ts_tablet_manager.cc:1875] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.792198 10335 ts_tablet_manager.cc:1875] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.792672 10565 ts_tablet_manager.cc:1888] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.792739 10565 log.cc:1196] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/37bc46947720491b9774cae8c450457c
I20240503 07:59:12.793002 10565 ts_tablet_manager.cc:1909] T 37bc46947720491b9774cae8c450457c P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:12.794450 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 37bc46947720491b9774cae8c450457c (table rebalancer_dynamic_table_new_name [id=b3da677241944a9bb4a3fa7febb7c2b3]) successfully deleted
I20240503 07:59:12.794939 10335 ts_tablet_manager.cc:1888] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.795092 10335 log.cc:1196] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/37bc46947720491b9774cae8c450457c
I20240503 07:59:12.795365 10335 ts_tablet_manager.cc:1909] T 37bc46947720491b9774cae8c450457c P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:12.795447 10564 ts_tablet_manager.cc:1888] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.795543 10564 log.cc:1196] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/37bc46947720491b9774cae8c450457c
I20240503 07:59:12.795789 10564 ts_tablet_manager.cc:1909] T 37bc46947720491b9774cae8c450457c P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:12.796353 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 37bc46947720491b9774cae8c450457c (table rebalancer_dynamic_table_new_name [id=b3da677241944a9bb4a3fa7febb7c2b3]) successfully deleted
I20240503 07:59:12.796514 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 37bc46947720491b9774cae8c450457c (table rebalancer_dynamic_table_new_name [id=b3da677241944a9bb4a3fa7febb7c2b3]) successfully deleted
I20240503 07:59:12.812449 7149 consensus_queue.cc:237] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:12.814177 7282 raft_consensus.cc:1261] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.814445 6886 raft_consensus.cc:1261] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.814730 10506 consensus_queue.cc:1035] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.814988 10158 consensus_queue.cc:1035] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:12.817883 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5390e70b0449449aa3af05112bd29797. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.817946 10158 raft_consensus.cc:2932] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:12.818513 6886 raft_consensus.cc:2932] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:12.820082 7282 raft_consensus.cc:2932] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:12.821151 6740 catalog_manager.cc:5484] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:12.823402 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:12.827926 9374 raft_consensus.cc:1203] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.831647 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 866c1bf17d444315bb3eb83a7a283b54 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=9b4298d7d24e485d91d7ae96bac125e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.831887 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 4737c826ebb64177a718d83b939c3e7b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=9b4298d7d24e485d91d7ae96bac125e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.831998 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866c1bf17d444315bb3eb83a7a283b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.832160 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4737c826ebb64177a718d83b939c3e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.832212 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 866c1bf17d444315bb3eb83a7a283b54 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=9b4298d7d24e485d91d7ae96bac125e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.832500 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866c1bf17d444315bb3eb83a7a283b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.835083 9616 tablet_service.cc:1467] Processing CreateTablet for tablet bb376275a94144f79dc88163fe1ee716 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=9b4298d7d24e485d91d7ae96bac125e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.835405 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb376275a94144f79dc88163fe1ee716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.835733 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 866c1bf17d444315bb3eb83a7a283b54 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=9b4298d7d24e485d91d7ae96bac125e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.836051 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866c1bf17d444315bb3eb83a7a283b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.836473 9485 tablet_service.cc:1467] Processing CreateTablet for tablet bb376275a94144f79dc88163fe1ee716 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=9b4298d7d24e485d91d7ae96bac125e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.836750 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb376275a94144f79dc88163fe1ee716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.837633 10591 tablet_bootstrap.cc:492] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:12.838063 6869 tablet_service.cc:1467] Processing CreateTablet for tablet bb376275a94144f79dc88163fe1ee716 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=9b4298d7d24e485d91d7ae96bac125e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.838383 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb376275a94144f79dc88163fe1ee716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.838688 10591 tablet_bootstrap.cc:654] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.838930 10594 tablet_bootstrap.cc:492] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:12.840080 10594 tablet_bootstrap.cc:654] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.840497 9374 raft_consensus.cc:1203] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.840831 10596 tablet_bootstrap.cc:492] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:12.841022 9353 tablet_service.cc:1467] Processing CreateTablet for tablet 4737c826ebb64177a718d83b939c3e7b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=9b4298d7d24e485d91d7ae96bac125e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.841280 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4737c826ebb64177a718d83b939c3e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.842088 10596 tablet_bootstrap.cc:654] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.845082 10599 tablet_bootstrap.cc:492] T bb376275a94144f79dc88163fe1ee716 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:12.845082 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 4737c826ebb64177a718d83b939c3e7b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=9b4298d7d24e485d91d7ae96bac125e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.845399 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4737c826ebb64177a718d83b939c3e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.845927 10599 tablet_bootstrap.cc:654] T bb376275a94144f79dc88163fe1ee716 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.846273 10593 tablet_bootstrap.cc:492] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:12.847256 10593 tablet_bootstrap.cc:654] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.847923 10592 tablet_bootstrap.cc:492] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:12.848830 10591 tablet_bootstrap.cc:492] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:12.848920 10592 tablet_bootstrap.cc:654] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.849021 10591 ts_tablet_manager.cc:1367] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:12.849537 10591 raft_consensus.cc:348] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.849751 10591 raft_consensus.cc:374] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.849854 10591 raft_consensus.cc:729] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.850178 10594 tablet_bootstrap.cc:492] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:12.850255 10592 tablet_bootstrap.cc:492] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:12.850279 10594 ts_tablet_manager.cc:1367] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:12.850086 10591 consensus_queue.cc:260] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.850394 10592 ts_tablet_manager.cc:1367] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:12.850625 10591 ts_tablet_manager.cc:1398] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:12.850782 10591 tablet_bootstrap.cc:492] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:12.850790 10594 raft_consensus.cc:348] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:12.850903 10594 raft_consensus.cc:374] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.850930 10594 raft_consensus.cc:729] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.851047 10594 consensus_queue.cc:260] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:12.851022 10592 raft_consensus.cc:348] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:12.851169 10592 raft_consensus.cc:374] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.851207 10592 raft_consensus.cc:729] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.851310 10594 ts_tablet_manager.cc:1398] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.851365 10592 consensus_queue.cc:260] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:12.851469 10594 tablet_bootstrap.cc:492] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:12.851634 10592 ts_tablet_manager.cc:1398] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.851728 10591 tablet_bootstrap.cc:654] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.851804 10592 tablet_bootstrap.cc:492] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:12.852383 10594 tablet_bootstrap.cc:654] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.852980 10593 tablet_bootstrap.cc:492] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:12.853071 10593 ts_tablet_manager.cc:1367] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:12.853163 9636 raft_consensus.cc:1203] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.853520 10593 raft_consensus.cc:348] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.853627 10593 raft_consensus.cc:374] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.853658 10593 raft_consensus.cc:729] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.853803 10593 consensus_queue.cc:260] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.854310 10593 ts_tablet_manager.cc:1398] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.855227 9837 raft_consensus.cc:482] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:12.855324 9837 raft_consensus.cc:504] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.855728 9837 leader_election.cc:288] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:12.856108 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb376275a94144f79dc88163fe1ee716" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:12.856525 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb376275a94144f79dc88163fe1ee716" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
W20240503 07:59:12.857080 9572 leader_election.cc:341] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Illegal state: must be running to vote when last-logged opid is not known
W20240503 07:59:12.857460 9569 leader_election.cc:341] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:12.857543 9569 leader_election.cc:302] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 159177b3caeb434baa98e04136d050b2; no voters: c4218904dd7e42d2be95b901a6e5db92, cb73994515af4880b05383e78368bb4a
I20240503 07:59:12.857674 9837 raft_consensus.cc:2726] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:12.858381 10592 tablet_bootstrap.cc:654] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.859025 10594 tablet_bootstrap.cc:492] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:12.859144 10594 ts_tablet_manager.cc:1367] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:12.859706 10596 tablet_bootstrap.cc:492] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:12.859640 10594 raft_consensus.cc:348] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.859767 10594 raft_consensus.cc:374] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.859795 10594 raft_consensus.cc:729] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.859812 10596 ts_tablet_manager.cc:1367] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 07:59:12.859916 10594 consensus_queue.cc:260] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.860191 10594 ts_tablet_manager.cc:1398] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.860320 10596 raft_consensus.cc:348] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.860491 10596 raft_consensus.cc:374] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.860570 10596 raft_consensus.cc:729] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.860767 10596 consensus_queue.cc:260] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.861068 10596 ts_tablet_manager.cc:1398] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.861604 10592 tablet_bootstrap.cc:492] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:12.861714 10592 ts_tablet_manager.cc:1367] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:12.862260 10592 raft_consensus.cc:348] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.862452 10592 raft_consensus.cc:374] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.862509 10592 raft_consensus.cc:729] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.862663 10592 consensus_queue.cc:260] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:12.863066 10592 ts_tablet_manager.cc:1398] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.864321 10599 tablet_bootstrap.cc:492] T bb376275a94144f79dc88163fe1ee716 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:12.864449 10599 ts_tablet_manager.cc:1367] T bb376275a94144f79dc88163fe1ee716 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 07:59:12.864969 10599 raft_consensus.cc:348] T bb376275a94144f79dc88163fe1ee716 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.865087 10599 raft_consensus.cc:374] T bb376275a94144f79dc88163fe1ee716 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.865118 10599 raft_consensus.cc:729] T bb376275a94144f79dc88163fe1ee716 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.865270 10599 consensus_queue.cc:260] T bb376275a94144f79dc88163fe1ee716 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:12.865568 10599 ts_tablet_manager.cc:1398] T bb376275a94144f79dc88163fe1ee716 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.867101 10591 tablet_bootstrap.cc:492] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:12.867193 10591 ts_tablet_manager.cc:1367] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.001s
I20240503 07:59:12.867682 10591 raft_consensus.cc:348] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:12.867810 10591 raft_consensus.cc:374] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.867830 10591 raft_consensus.cc:729] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.867945 10591 consensus_queue.cc:260] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:12.868214 10591 ts_tablet_manager.cc:1398] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:12.889740 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:12.898111 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 9875f6a31e704b76a61fba3f8fc3739f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3963170cce0945b08fbb27981b7dc044]), partition=
I20240503 07:59:12.898486 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9875f6a31e704b76a61fba3f8fc3739f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.900435 10593 tablet_bootstrap.cc:492] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:12.900872 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 9875f6a31e704b76a61fba3f8fc3739f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3963170cce0945b08fbb27981b7dc044]), partition=
I20240503 07:59:12.901154 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9875f6a31e704b76a61fba3f8fc3739f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.901454 10593 tablet_bootstrap.cc:654] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.902673 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 9875f6a31e704b76a61fba3f8fc3739f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3963170cce0945b08fbb27981b7dc044]), partition=
I20240503 07:59:12.902925 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9875f6a31e704b76a61fba3f8fc3739f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.903296 10592 tablet_bootstrap.cc:492] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:12.904436 10592 tablet_bootstrap.cc:654] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.909811 10593 tablet_bootstrap.cc:492] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:12.909942 10593 ts_tablet_manager.cc:1367] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:12.910495 10593 raft_consensus.cc:348] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.910636 10593 raft_consensus.cc:374] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.910670 10593 raft_consensus.cc:729] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.910871 10593 consensus_queue.cc:260] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.911394 10593 ts_tablet_manager.cc:1398] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 07:59:12.915377 10637 tablet_bootstrap.cc:492] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:12.916484 10637 tablet_bootstrap.cc:654] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.917166 10358 raft_consensus.cc:482] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:12.917294 10358 raft_consensus.cc:504] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:12.917671 10592 tablet_bootstrap.cc:492] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:12.917805 10592 ts_tablet_manager.cc:1367] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:12.917809 10358 leader_election.cc:288] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:12.918407 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866c1bf17d444315bb3eb83a7a283b54" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:12.918521 9767 raft_consensus.cc:2454] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:12.918606 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866c1bf17d444315bb3eb83a7a283b54" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:12.918746 7278 raft_consensus.cc:2454] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:12.918851 9310 leader_election.cc:302] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:12.919030 10358 raft_consensus.cc:2781] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:12.919077 10358 raft_consensus.cc:482] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:12.919111 10358 raft_consensus.cc:3037] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.920035 10358 raft_consensus.cc:504] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:12.920418 10358 leader_election.cc:288] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:12.918332 10592 raft_consensus.cc:348] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.920790 10592 raft_consensus.cc:374] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.920850 10592 raft_consensus.cc:729] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.920997 10592 consensus_queue.cc:260] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.921097 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866c1bf17d444315bb3eb83a7a283b54" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:12.921223 9767 raft_consensus.cc:3037] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.921514 10592 ts_tablet_manager.cc:1398] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:12.921823 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866c1bf17d444315bb3eb83a7a283b54" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:12.921938 7278 raft_consensus.cc:3037] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.922331 9767 raft_consensus.cc:2454] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:12.922801 7278 raft_consensus.cc:2454] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:12.924235 9310 leader_election.cc:302] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:12.924443 10358 raft_consensus.cc:2781] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:12.924540 10358 raft_consensus.cc:686] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:12.924685 10358 consensus_queue.cc:237] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:12.928257 9636 raft_consensus.cc:1203] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.928501 6745 catalog_manager.cc:5484] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:12.931845 10637 tablet_bootstrap.cc:492] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:12.932302 10637 ts_tablet_manager.cc:1367] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 07:59:12.932940 10637 raft_consensus.cc:348] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.933156 10637 raft_consensus.cc:374] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.933238 10637 raft_consensus.cc:729] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.933439 10637 consensus_queue.cc:260] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:12.934581 10637 ts_tablet_manager.cc:1398] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:12.948097 9374 raft_consensus.cc:1203] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.988031 10424 ts_tablet_manager.cc:902] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:12.988606 10424 tablet_copy_client.cc:308] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46383
I20240503 07:59:12.989154 7040 tablet_copy_service.cc:140] P 9c085a3241f14cb7ba1332b508f4b57d: Received BeginTabletCopySession request for tablet 7b424a8b7bba44b098731f8ee18e8c9b from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:49881)
I20240503 07:59:12.989290 7040 tablet_copy_service.cc:161] P 9c085a3241f14cb7ba1332b508f4b57d: Beginning new tablet copy session on tablet 7b424a8b7bba44b098731f8ee18e8c9b from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:49881: session id = 159177b3caeb434baa98e04136d050b2-7b424a8b7bba44b098731f8ee18e8c9b
I20240503 07:59:12.990304 7040 tablet_copy_source_session.cc:205] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.991291 10424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b424a8b7bba44b098731f8ee18e8c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.993312 10424 tablet_copy_client.cc:783] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.993472 10424 tablet_copy_client.cc:647] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.994709 10424 tablet_copy_client.cc:523] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.996117 10424 tablet_bootstrap.cc:492] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:13.005075 10532 raft_consensus.cc:482] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:13.005206 10532 raft_consensus.cc:504] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.005771 10532 leader_election.cc:288] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:13.006385 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9875f6a31e704b76a61fba3f8fc3739f" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:13.006552 7149 raft_consensus.cc:2454] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:13.007339 9703 leader_election.cc:302] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:13.007642 10532 raft_consensus.cc:2781] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:13.007699 10532 raft_consensus.cc:482] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:13.007730 10532 raft_consensus.cc:3037] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.008867 10532 raft_consensus.cc:504] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.009243 10532 leader_election.cc:288] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:13.009799 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9875f6a31e704b76a61fba3f8fc3739f" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:13.009938 7149 raft_consensus.cc:3037] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.010983 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9875f6a31e704b76a61fba3f8fc3739f" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:13.011026 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9875f6a31e704b76a61fba3f8fc3739f" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:13.011094 7149 raft_consensus.cc:2454] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:13.011189 9636 raft_consensus.cc:2454] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:13.011698 9703 leader_election.cc:302] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters: 159177b3caeb434baa98e04136d050b2
I20240503 07:59:13.011910 10532 raft_consensus.cc:2781] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:13.012007 10532 raft_consensus.cc:686] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:13.012140 10532 consensus_queue.cc:237] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.013999 6740 catalog_manager.cc:5484] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:13.021757 10424 tablet_bootstrap.cc:492] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.022074 10424 tablet_bootstrap.cc:492] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:13.022269 10424 ts_tablet_manager.cc:1367] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.000s
I20240503 07:59:13.022763 10424 raft_consensus.cc:348] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.022895 10424 raft_consensus.cc:729] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.023026 10424 consensus_queue.cc:260] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.023327 10424 ts_tablet_manager.cc:1398] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 07:59:13.023948 7040 tablet_copy_service.cc:342] P 9c085a3241f14cb7ba1332b508f4b57d: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-7b424a8b7bba44b098731f8ee18e8c9b received from {username='slave'} at 127.0.99.135:49881
I20240503 07:59:13.024045 7040 tablet_copy_service.cc:433] P 9c085a3241f14cb7ba1332b508f4b57d: ending tablet copy session 159177b3caeb434baa98e04136d050b2-7b424a8b7bba44b098731f8ee18e8c9b on tablet 7b424a8b7bba44b098731f8ee18e8c9b with peer 159177b3caeb434baa98e04136d050b2
I20240503 07:59:13.035665 10423 ts_tablet_manager.cc:902] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:13.036181 10423 tablet_copy_client.cc:308] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:13.036741 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet b916e89edaf046b99767e5cca531bb8b from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:56405)
I20240503 07:59:13.036877 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet b916e89edaf046b99767e5cca531bb8b from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:56405: session id = 3befbb2d206f4be89d3e07f38837374b-b916e89edaf046b99767e5cca531bb8b
I20240503 07:59:13.037705 7302 tablet_copy_source_session.cc:205] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.038651 10423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b916e89edaf046b99767e5cca531bb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.040858 10423 tablet_copy_client.cc:783] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.041720 10423 tablet_copy_client.cc:647] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.043119 10423 tablet_copy_client.cc:523] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.044275 10423 tablet_bootstrap.cc:492] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:13.063051 10423 tablet_bootstrap.cc:492] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.063308 10423 tablet_bootstrap.cc:492] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:13.063460 10423 ts_tablet_manager.cc:1367] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.004s
I20240503 07:59:13.063887 10423 raft_consensus.cc:348] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:13.064002 10423 raft_consensus.cc:729] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.064114 10423 consensus_queue.cc:260] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:13.064383 10423 ts_tablet_manager.cc:1398] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:13.064963 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-b916e89edaf046b99767e5cca531bb8b received from {username='slave'} at 127.0.99.133:56405
I20240503 07:59:13.065044 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-b916e89edaf046b99767e5cca531bb8b on tablet b916e89edaf046b99767e5cca531bb8b with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:13.077942 9899 raft_consensus.cc:482] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:13.078102 9899 raft_consensus.cc:504] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:13.078598 9899 leader_election.cc:288] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:13.079062 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb376275a94144f79dc88163fe1ee716" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:13.079080 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb376275a94144f79dc88163fe1ee716" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:13.079202 9504 raft_consensus.cc:2454] T bb376275a94144f79dc88163fe1ee716 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:13.079229 9636 raft_consensus.cc:2454] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:13.079546 6825 leader_election.cc:302] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:13.079792 9899 raft_consensus.cc:2781] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:13.079854 9899 raft_consensus.cc:482] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:13.079888 9899 raft_consensus.cc:3037] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.081051 9899 raft_consensus.cc:504] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:13.081446 9899 leader_election.cc:288] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:13.081849 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb376275a94144f79dc88163fe1ee716" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:13.081992 9636 raft_consensus.cc:3037] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.082116 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb376275a94144f79dc88163fe1ee716" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:13.082244 9504 raft_consensus.cc:3037] T bb376275a94144f79dc88163fe1ee716 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.082616 9767 raft_consensus.cc:1203] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.083163 9636 raft_consensus.cc:2454] T bb376275a94144f79dc88163fe1ee716 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:13.083348 9504 raft_consensus.cc:2454] T bb376275a94144f79dc88163fe1ee716 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:13.083477 6822 leader_election.cc:302] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:13.084144 9899 raft_consensus.cc:2781] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:13.084300 9899 raft_consensus.cc:686] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:13.084447 9899 consensus_queue.cc:237] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:13.086390 6740 catalog_manager.cc:5484] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:13.118363 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:13.122808 9636 raft_consensus.cc:3037] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.122853 7149 raft_consensus.cc:1261] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:13.123349 10166 consensus_queue.cc:1035] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.124081 9636 raft_consensus.cc:1261] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:13.124639 10166 consensus_queue.cc:1035] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.126859 6740 catalog_manager.cc:5576] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317) does not have the latest schema for tablet 9875f6a31e704b76a61fba3f8fc3739f (table rebalancer_dynamic_table [id=3963170cce0945b08fbb27981b7dc044]). Expected version 1 got 0
I20240503 07:59:13.128948 10122 tablet.cc:1722] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:13.129329 10122 tablet.cc:1620] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:13.129374 10413 tablet.cc:1722] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:13.129686 10413 tablet.cc:1620] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:13.130457 6746 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 9875f6a31e704b76a61fba3f8fc3739f (table rebalancer_dynamic_table [id=3963170cce0945b08fbb27981b7dc044]). Expected version 1 got 0
I20240503 07:59:13.130827 6742 catalog_manager.cc:5576] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317) does not have the latest schema for tablet 9875f6a31e704b76a61fba3f8fc3739f (table rebalancer_dynamic_table [id=3963170cce0945b08fbb27981b7dc044]). Expected version 1 got 0
I20240503 07:59:13.133021 10656 tablet.cc:1722] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:13.133488 10656 tablet.cc:1620] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:13.133924 6746 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 9875f6a31e704b76a61fba3f8fc3739f (table rebalancer_dynamic_table [id=3963170cce0945b08fbb27981b7dc044]). Expected version 1 got 0
I20240503 07:59:13.135648 6742 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=3963170cce0945b08fbb27981b7dc044] alter complete (version 1)
I20240503 07:59:13.141467 10657 ts_tablet_manager.cc:902] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:13.142174 10657 tablet_copy_client.cc:308] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46383
I20240503 07:59:13.145570 7040 tablet_copy_service.cc:140] P 9c085a3241f14cb7ba1332b508f4b57d: Received BeginTabletCopySession request for tablet 82fc9f4263244c6fa2fb39543230eb87 from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:41329)
I20240503 07:59:13.145685 7040 tablet_copy_service.cc:161] P 9c085a3241f14cb7ba1332b508f4b57d: Beginning new tablet copy session on tablet 82fc9f4263244c6fa2fb39543230eb87 from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:41329: session id = cb73994515af4880b05383e78368bb4a-82fc9f4263244c6fa2fb39543230eb87
I20240503 07:59:13.146529 7040 tablet_copy_source_session.cc:205] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.147415 10657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fc9f4263244c6fa2fb39543230eb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.149471 10657 tablet_copy_client.cc:783] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.149674 10657 tablet_copy_client.cc:647] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.150928 10657 tablet_copy_client.cc:523] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.152273 10657 tablet_bootstrap.cc:492] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:13.162747 10489 raft_consensus.cc:482] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:13.162880 10489 raft_consensus.cc:504] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.163334 10489 leader_election.cc:288] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:13.163844 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4737c826ebb64177a718d83b939c3e7b" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:13.163988 9767 raft_consensus.cc:2454] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:13.164561 7218 leader_election.cc:302] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:13.164852 10489 raft_consensus.cc:2781] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:13.164908 10489 raft_consensus.cc:482] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:13.164942 10489 raft_consensus.cc:3037] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.164875 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4737c826ebb64177a718d83b939c3e7b" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:13.165023 9374 raft_consensus.cc:2454] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:13.166275 10489 raft_consensus.cc:504] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.166703 10489 leader_election.cc:288] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:13.167055 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4737c826ebb64177a718d83b939c3e7b" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:13.167060 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4737c826ebb64177a718d83b939c3e7b" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:13.167188 9374 raft_consensus.cc:3037] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.167186 9767 raft_consensus.cc:3037] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.168121 9767 raft_consensus.cc:2454] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:13.168155 9374 raft_consensus.cc:2454] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:13.168423 7218 leader_election.cc:302] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:13.168594 10489 raft_consensus.cc:2781] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:13.168736 10489 raft_consensus.cc:686] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:13.168901 10489 consensus_queue.cc:237] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.171262 6746 catalog_manager.cc:5484] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:13.172048 9767 raft_consensus.cc:1203] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.176043 10657 tablet_bootstrap.cc:492] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.176457 10657 tablet_bootstrap.cc:492] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:13.176681 10657 ts_tablet_manager.cc:1367] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.024s user 0.025s sys 0.000s
I20240503 07:59:13.177331 10657 raft_consensus.cc:348] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:13.177516 10657 raft_consensus.cc:729] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.177699 10657 consensus_queue.cc:260] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:13.178025 10657 ts_tablet_manager.cc:1398] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.178603 7040 tablet_copy_service.cc:342] P 9c085a3241f14cb7ba1332b508f4b57d: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-82fc9f4263244c6fa2fb39543230eb87 received from {username='slave'} at 127.0.99.134:41329
I20240503 07:59:13.178712 7040 tablet_copy_service.cc:433] P 9c085a3241f14cb7ba1332b508f4b57d: ending tablet copy session cb73994515af4880b05383e78368bb4a-82fc9f4263244c6fa2fb39543230eb87 on tablet 82fc9f4263244c6fa2fb39543230eb87 with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:13.183370 10372 ts_tablet_manager.cc:902] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:13.183966 10372 tablet_copy_client.cc:308] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:13.184595 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 7e8e04c50d7147af903d7113df3d9744 from peer a372e98a670f49e9adca46ab5c836c4a ({username='slave'} at 127.0.99.136:53081)
I20240503 07:59:13.184718 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 7e8e04c50d7147af903d7113df3d9744 from peer a372e98a670f49e9adca46ab5c836c4a at {username='slave'} at 127.0.99.136:53081: session id = a372e98a670f49e9adca46ab5c836c4a-7e8e04c50d7147af903d7113df3d9744
I20240503 07:59:13.185751 7171 tablet_copy_source_session.cc:205] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.186964 10372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8e04c50d7147af903d7113df3d9744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.188995 10372 tablet_copy_client.cc:783] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.189177 10372 tablet_copy_client.cc:647] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.189736 9504 raft_consensus.cc:1203] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.190984 6746 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 07:59:13.191136 6746 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 07:59:13.191324 10372 tablet_copy_client.cc:523] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.192781 10372 tablet_bootstrap.cc:492] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:13.195881 6746 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet fccdd597c5b54f41bb846d933a1e9a84
I20240503 07:59:13.196239 6746 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 91af7b042df146e980638f0f79bde295
I20240503 07:59:13.196471 6746 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 4 replicas of tablet b916e89edaf046b99767e5cca531bb8b
I20240503 07:59:13.196875 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet fccdd597c5b54f41bb846d933a1e9a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:13.197219 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet fccdd597c5b54f41bb846d933a1e9a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:13.197281 10336 tablet_replica.cc:330] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:13.197427 10512 tablet_replica.cc:330] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:13.197428 10336 raft_consensus.cc:2229] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.197571 10512 raft_consensus.cc:2229] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.197582 10336 raft_consensus.cc:2258] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.197718 10512 raft_consensus.cc:2258] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.197747 9353 tablet_service.cc:1514] Processing DeleteTablet for tablet b916e89edaf046b99767e5cca531bb8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:13.198038 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet b916e89edaf046b99767e5cca531bb8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:13.198159 10336 ts_tablet_manager.cc:1875] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.198282 9957 tablet_replica.cc:330] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:13.198267 10512 ts_tablet_manager.cc:1875] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.198448 9957 raft_consensus.cc:2229] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.198611 9957 raft_consensus.cc:2258] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.200646 10512 ts_tablet_manager.cc:1888] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.200656 10336 ts_tablet_manager.cc:1888] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.200716 10512 log.cc:1196] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/fccdd597c5b54f41bb846d933a1e9a84
I20240503 07:59:13.200721 10336 log.cc:1196] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/fccdd597c5b54f41bb846d933a1e9a84
I20240503 07:59:13.200978 10512 ts_tablet_manager.cc:1909] T fccdd597c5b54f41bb846d933a1e9a84 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:13.200978 10336 ts_tablet_manager.cc:1909] T fccdd597c5b54f41bb846d933a1e9a84 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:13.201195 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet fccdd597c5b54f41bb846d933a1e9a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:13.201336 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet 91af7b042df146e980638f0f79bde295 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:13.201464 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 91af7b042df146e980638f0f79bde295 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:13.201627 10335 tablet_replica.cc:330] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:13.201782 10335 raft_consensus.cc:2229] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.201932 10335 raft_consensus.cc:2258] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.202227 7130 tablet_service.cc:1514] Processing DeleteTablet for tablet b916e89edaf046b99767e5cca531bb8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:13.202616 10335 ts_tablet_manager.cc:1875] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.202927 9957 ts_tablet_manager.cc:1875] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.204905 10335 ts_tablet_manager.cc:1888] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.204972 10335 log.cc:1196] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/91af7b042df146e980638f0f79bde295
I20240503 07:59:13.205219 10335 ts_tablet_manager.cc:1909] T 91af7b042df146e980638f0f79bde295 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:13.205296 9957 ts_tablet_manager.cc:1888] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.205370 9957 log.cc:1196] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b916e89edaf046b99767e5cca531bb8b
I20240503 07:59:13.205605 9957 ts_tablet_manager.cc:1909] T b916e89edaf046b99767e5cca531bb8b P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:13.205806 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 91af7b042df146e980638f0f79bde295 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:13.197243 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet b916e89edaf046b99767e5cca531bb8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:13.208120 10336 tablet_replica.cc:330] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:13.208119 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet fccdd597c5b54f41bb846d933a1e9a84 (table rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]) successfully deleted
I20240503 07:59:13.208277 10336 raft_consensus.cc:2229] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.208487 10336 raft_consensus.cc:2258] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.208627 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet b916e89edaf046b99767e5cca531bb8b (table rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]) successfully deleted
I20240503 07:59:13.208909 9957 tablet_replica.cc:330] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:13.209026 10336 ts_tablet_manager.cc:1875] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.209046 9957 raft_consensus.cc:2229] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Raft consensus shutting down.
W20240503 07:59:13.209190 7215 consensus_peers.cc:485] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc -> Peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Couldn't send request to peer c4218904dd7e42d2be95b901a6e5db92. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.209242 9957 raft_consensus.cc:2258] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.209746 10512 tablet_replica.cc:330] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:13.209810 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet fccdd597c5b54f41bb846d933a1e9a84 (table rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]) successfully deleted
I20240503 07:59:13.209851 9957 ts_tablet_manager.cc:1875] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.210228 10335 tablet_replica.cc:330] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:13.210304 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 91af7b042df146e980638f0f79bde295 (table rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]) successfully deleted
I20240503 07:59:13.210486 10512 raft_consensus.cc:2229] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.210661 10335 raft_consensus.cc:2229] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.211036 10335 raft_consensus.cc:2258] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.211160 10512 raft_consensus.cc:2258] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.212088 10512 ts_tablet_manager.cc:1875] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.212126 10335 ts_tablet_manager.cc:1875] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.215003 10335 ts_tablet_manager.cc:1888] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.215013 10512 ts_tablet_manager.cc:1888] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.215091 10335 log.cc:1196] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b916e89edaf046b99767e5cca531bb8b
I20240503 07:59:13.215222 10512 log.cc:1196] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b916e89edaf046b99767e5cca531bb8b
I20240503 07:59:13.215546 10512 ts_tablet_manager.cc:1909] T b916e89edaf046b99767e5cca531bb8b P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:13.215901 10336 ts_tablet_manager.cc:1888] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.215987 10336 log.cc:1196] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/91af7b042df146e980638f0f79bde295
I20240503 07:59:13.216223 10336 ts_tablet_manager.cc:1909] T 91af7b042df146e980638f0f79bde295 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:13.216965 10335 ts_tablet_manager.cc:1909] T b916e89edaf046b99767e5cca531bb8b P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:13.218065 10670 tablet_replica.cc:330] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:13.218081 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 91af7b042df146e980638f0f79bde295 (table rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]) successfully deleted
I20240503 07:59:13.218374 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet b916e89edaf046b99767e5cca531bb8b (table rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]) successfully deleted
I20240503 07:59:13.218381 10670 raft_consensus.cc:2229] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.218585 10670 raft_consensus.cc:2258] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.219008 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet b916e89edaf046b99767e5cca531bb8b (table rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]) successfully deleted
I20240503 07:59:13.221339 10670 ts_tablet_manager.cc:1875] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.224664 10670 ts_tablet_manager.cc:1888] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.224843 10670 log.cc:1196] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/b916e89edaf046b99767e5cca531bb8b
I20240503 07:59:13.225185 10670 ts_tablet_manager.cc:1909] T b916e89edaf046b99767e5cca531bb8b P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:13.225971 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet b916e89edaf046b99767e5cca531bb8b (table rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]) successfully deleted
I20240503 07:59:13.226090 9616 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:13.226879 9353 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:13.231405 9957 ts_tablet_manager.cc:1888] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.231626 9957 log.cc:1196] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/fccdd597c5b54f41bb846d933a1e9a84
I20240503 07:59:13.231943 9957 ts_tablet_manager.cc:1909] T fccdd597c5b54f41bb846d933a1e9a84 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:13.232862 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet fccdd597c5b54f41bb846d933a1e9a84 (table rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]) successfully deleted
I20240503 07:59:13.233234 9957 tablet_replica.cc:330] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:13.233486 9957 raft_consensus.cc:2229] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.233736 9957 raft_consensus.cc:2258] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.234485 9957 ts_tablet_manager.cc:1875] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.236537 7261 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:13.237115 9957 ts_tablet_manager.cc:1888] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.237311 9957 log.cc:1196] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/91af7b042df146e980638f0f79bde295
I20240503 07:59:13.237855 6998 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:13.240726 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:13.243410 9957 ts_tablet_manager.cc:1909] T 91af7b042df146e980638f0f79bde295 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:13.247346 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 91af7b042df146e980638f0f79bde295 (table rebalance_test_table_8 [id=a50a777c731c4a7dac4fe35f5d3be668]) successfully deleted
I20240503 07:59:13.250005 10122 tablet.cc:1722] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:13.250397 10122 tablet.cc:1620] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:13.253683 10656 tablet.cc:1722] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:13.254140 10656 tablet.cc:1620] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:13.255923 10413 tablet.cc:1722] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:13.256302 10413 tablet.cc:1620] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:13.261919 10372 tablet_bootstrap.cc:492] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.263425 10372 tablet_bootstrap.cc:492] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap complete.
I20240503 07:59:13.263706 10372 ts_tablet_manager.cc:1367] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.071s user 0.027s sys 0.000s
I20240503 07:59:13.268429 6742 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=3963170cce0945b08fbb27981b7dc044] alter complete (version 2)
I20240503 07:59:13.269500 10372 raft_consensus.cc:348] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.269798 10372 raft_consensus.cc:729] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.270411 10372 consensus_queue.cc:260] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.273674 10372 ts_tablet_manager.cc:1398] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240503 07:59:13.274804 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session a372e98a670f49e9adca46ab5c836c4a-7e8e04c50d7147af903d7113df3d9744 received from {username='slave'} at 127.0.99.136:53081
I20240503 07:59:13.274897 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session a372e98a670f49e9adca46ab5c836c4a-7e8e04c50d7147af903d7113df3d9744 on tablet 7e8e04c50d7147af903d7113df3d9744 with peer a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:13.277086 9485 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:13.283063 10424 ts_tablet_manager.cc:902] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:13.283565 10424 tablet_copy_client.cc:308] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46383
I20240503 07:59:13.284250 7040 tablet_copy_service.cc:140] P 9c085a3241f14cb7ba1332b508f4b57d: Received BeginTabletCopySession request for tablet 31c76213f3ec451084ae7b0551cd7b48 from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:49881)
I20240503 07:59:13.284384 7040 tablet_copy_service.cc:161] P 9c085a3241f14cb7ba1332b508f4b57d: Beginning new tablet copy session on tablet 31c76213f3ec451084ae7b0551cd7b48 from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:49881: session id = 159177b3caeb434baa98e04136d050b2-31c76213f3ec451084ae7b0551cd7b48
I20240503 07:59:13.285380 7040 tablet_copy_source_session.cc:205] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.285843 6868 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:13.287894 10424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c76213f3ec451084ae7b0551cd7b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.297434 10424 tablet_copy_client.cc:783] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.297811 10424 tablet_copy_client.cc:647] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.302417 10424 tablet_copy_client.cc:523] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.303867 10424 tablet_bootstrap.cc:492] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:13.305593 9746 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:13.306509 7130 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:13.333488 10358 consensus_queue.cc:1035] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.338940 10424 tablet_bootstrap.cc:492] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.339257 10424 tablet_bootstrap.cc:492] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:13.339473 10424 ts_tablet_manager.cc:1367] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.036s user 0.025s sys 0.000s
I20240503 07:59:13.340005 10424 raft_consensus.cc:348] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.340150 10424 raft_consensus.cc:729] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.340301 10424 consensus_queue.cc:260] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.340587 10424 ts_tablet_manager.cc:1398] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:13.341920 7040 tablet_copy_service.cc:342] P 9c085a3241f14cb7ba1332b508f4b57d: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-31c76213f3ec451084ae7b0551cd7b48 received from {username='slave'} at 127.0.99.135:49881
I20240503 07:59:13.342012 7040 tablet_copy_service.cc:433] P 9c085a3241f14cb7ba1332b508f4b57d: ending tablet copy session 159177b3caeb434baa98e04136d050b2-31c76213f3ec451084ae7b0551cd7b48 on tablet 31c76213f3ec451084ae7b0551cd7b48 with peer 159177b3caeb434baa98e04136d050b2
I20240503 07:59:13.344337 9985 consensus_queue.cc:1035] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.373498 7150 consensus_queue.cc:237] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:13.375427 7278 raft_consensus.cc:1261] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:13.375519 7087 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5365969434f142b8a5757a9a73bc109a. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.375872 7019 raft_consensus.cc:1261] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:13.376273 10158 consensus_queue.cc:1035] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:13.376284 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:13.376493 10505 consensus_queue.cc:1035] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:13.379233 10158 raft_consensus.cc:2932] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:13.382218 7278 raft_consensus.cc:2932] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:13.382539 7019 raft_consensus.cc:2932] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:13.383585 10122 tablet.cc:1722] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:13.384465 10656 tablet.cc:1722] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:13.386881 6740 catalog_manager.cc:5484] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:13.391490 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=3963170cce0945b08fbb27981b7dc044] alter complete (version 3)
I20240503 07:59:13.394017 10413 tablet.cc:1722] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:13.416671 7149 consensus_queue.cc:237] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.418577 7019 raft_consensus.cc:1261] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:13.418913 7087 consensus_peers.cc:485] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Couldn't send request to peer a372e98a670f49e9adca46ab5c836c4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16c06ab5a3e54ed2910afa50069810cc. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.419016 7278 raft_consensus.cc:1261] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:13.419030 10713 consensus_queue.cc:1035] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:13.419454 10713 consensus_queue.cc:1035] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:13.423862 10713 raft_consensus.cc:2932] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:13.424439 7278 raft_consensus.cc:2932] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:13.426965 7019 raft_consensus.cc:2932] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:13.427284 6741 catalog_manager.cc:5484] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:13.430725 10423 ts_tablet_manager.cc:902] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:13.431416 10423 tablet_copy_client.cc:308] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:13.436772 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 5390e70b0449449aa3af05112bd29797 from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:52953)
I20240503 07:59:13.436918 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 5390e70b0449449aa3af05112bd29797 from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:52953: session id = 3befbb2d206f4be89d3e07f38837374b-5390e70b0449449aa3af05112bd29797
I20240503 07:59:13.438025 7171 tablet_copy_source_session.cc:205] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.439463 10423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5390e70b0449449aa3af05112bd29797. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.442721 10423 tablet_copy_client.cc:783] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.443009 10423 tablet_copy_client.cc:647] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.444628 10423 tablet_copy_client.cc:523] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.446305 10423 tablet_bootstrap.cc:492] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:13.470916 10423 tablet_bootstrap.cc:492] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.471228 10423 tablet_bootstrap.cc:492] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:13.471433 10423 ts_tablet_manager.cc:1367] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.005s
I20240503 07:59:13.471899 10423 raft_consensus.cc:348] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:13.472023 10423 raft_consensus.cc:729] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.472198 10423 consensus_queue.cc:260] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:13.472599 10423 ts_tablet_manager.cc:1398] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:13.473309 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-5390e70b0449449aa3af05112bd29797 received from {username='slave'} at 127.0.99.133:52953
I20240503 07:59:13.473394 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-5390e70b0449449aa3af05112bd29797 on tablet 5390e70b0449449aa3af05112bd29797 with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:13.495155 6741 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:13.495416 6741 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:13.498865 6741 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 9875f6a31e704b76a61fba3f8fc3739f
I20240503 07:59:13.499658 7130 tablet_service.cc:1514] Processing DeleteTablet for tablet 9875f6a31e704b76a61fba3f8fc3739f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:13.499691 9746 tablet_service.cc:1514] Processing DeleteTablet for tablet 9875f6a31e704b76a61fba3f8fc3739f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:13.499943 10335 tablet_replica.cc:330] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:13.500011 9616 tablet_service.cc:1514] Processing DeleteTablet for tablet 9875f6a31e704b76a61fba3f8fc3739f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:13.500061 10781 tablet_replica.cc:330] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:13.500095 10335 raft_consensus.cc:2229] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.500186 10781 raft_consensus.cc:2229] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.500227 10335 raft_consensus.cc:2258] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.500532 10781 raft_consensus.cc:2258] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.501327 10335 ts_tablet_manager.cc:1875] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.501339 10781 ts_tablet_manager.cc:1875] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.503648 10335 ts_tablet_manager.cc:1888] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.503729 10335 log.cc:1196] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/9875f6a31e704b76a61fba3f8fc3739f
I20240503 07:59:13.503957 10335 ts_tablet_manager.cc:1909] T 9875f6a31e704b76a61fba3f8fc3739f P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:13.504081 10781 ts_tablet_manager.cc:1888] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.504249 10781 log.cc:1196] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/9875f6a31e704b76a61fba3f8fc3739f
I20240503 07:59:13.504277 10782 tablet_replica.cc:330] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:13.504506 10782 raft_consensus.cc:2229] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.504582 10781 ts_tablet_manager.cc:1909] T 9875f6a31e704b76a61fba3f8fc3739f P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:13.504886 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 9875f6a31e704b76a61fba3f8fc3739f (table rebalancer_dynamic_table_new_name [id=3963170cce0945b08fbb27981b7dc044]) successfully deleted
I20240503 07:59:13.505461 10782 raft_consensus.cc:2258] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.506772 10782 ts_tablet_manager.cc:1875] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.511559 10782 ts_tablet_manager.cc:1888] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.511739 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 9875f6a31e704b76a61fba3f8fc3739f (table rebalancer_dynamic_table_new_name [id=3963170cce0945b08fbb27981b7dc044]) successfully deleted
I20240503 07:59:13.512053 10782 log.cc:1196] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/9875f6a31e704b76a61fba3f8fc3739f
I20240503 07:59:13.512423 10782 ts_tablet_manager.cc:1909] T 9875f6a31e704b76a61fba3f8fc3739f P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:13.513718 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 9875f6a31e704b76a61fba3f8fc3739f (table rebalancer_dynamic_table_new_name [id=3963170cce0945b08fbb27981b7dc044]) successfully deleted
I20240503 07:59:13.516240 9636 raft_consensus.cc:1203] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.535355 7130 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:13.540324 9353 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:13.541890 9616 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:13.542584 7261 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:13.545348 9504 raft_consensus.cc:1203] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.556510 6998 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:13.568481 6868 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:13.587198 9746 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:13.588204 9485 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:13.597388 9899 consensus_queue.cc:1035] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.601974 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:13.608381 9353 tablet_service.cc:1467] Processing CreateTablet for tablet 7d5f760fd6b64ab796c6d8355fa3616c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d6b27aaf354421c8436d0a1436d673c]), partition=
I20240503 07:59:13.608489 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 7d5f760fd6b64ab796c6d8355fa3616c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d6b27aaf354421c8436d0a1436d673c]), partition=
I20240503 07:59:13.608719 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d5f760fd6b64ab796c6d8355fa3616c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.608723 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d5f760fd6b64ab796c6d8355fa3616c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.610777 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 7d5f760fd6b64ab796c6d8355fa3616c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d6b27aaf354421c8436d0a1436d673c]), partition=
I20240503 07:59:13.611069 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d5f760fd6b64ab796c6d8355fa3616c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.614534 10795 tablet_bootstrap.cc:492] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:13.615756 10795 tablet_bootstrap.cc:654] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.617632 10795 tablet_bootstrap.cc:492] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:13.617839 10795 ts_tablet_manager.cc:1367] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:13.618722 10795 raft_consensus.cc:348] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.618986 10795 raft_consensus.cc:374] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.619077 10795 raft_consensus.cc:729] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.619522 10795 consensus_queue.cc:260] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.620167 9899 consensus_queue.cc:1035] T bb376275a94144f79dc88163fe1ee716 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.620741 10795 ts_tablet_manager.cc:1398] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:13.622043 10796 tablet_bootstrap.cc:492] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:13.624768 10796 tablet_bootstrap.cc:654] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.624810 10799 tablet_bootstrap.cc:492] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:13.626135 10796 tablet_bootstrap.cc:492] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:13.626284 10796 ts_tablet_manager.cc:1367] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:13.626991 10799 tablet_bootstrap.cc:654] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.627053 10796 raft_consensus.cc:348] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.627202 10796 raft_consensus.cc:374] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.627264 10796 raft_consensus.cc:729] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.627460 10796 consensus_queue.cc:260] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.627907 10796 ts_tablet_manager.cc:1398] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.654860 10799 tablet_bootstrap.cc:492] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:13.655027 10799 ts_tablet_manager.cc:1367] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240503 07:59:13.655629 10799 raft_consensus.cc:348] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.655750 10799 raft_consensus.cc:374] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.655778 10799 raft_consensus.cc:729] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.655921 10799 consensus_queue.cc:260] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.656185 10799 ts_tablet_manager.cc:1398] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:13.660099 10486 consensus_queue.cc:1035] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.663219 10486 consensus_queue.cc:1035] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.698135 6744 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 07:59:13.698297 6744 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 07:59:13.702723 6744 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet c86d285233c84f628d2ce4c531e3aef4
I20240503 07:59:13.703545 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet c86d285233c84f628d2ce4c531e3aef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:13.703694 7130 tablet_service.cc:1514] Processing DeleteTablet for tablet c86d285233c84f628d2ce4c531e3aef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:13.703810 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet c86d285233c84f628d2ce4c531e3aef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:13.703963 9957 tablet_replica.cc:330] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:13.704015 10336 tablet_replica.cc:330] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:13.704128 9957 raft_consensus.cc:2229] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.704174 10336 raft_consensus.cc:2229] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.704332 9957 raft_consensus.cc:2258] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.704394 10336 raft_consensus.cc:2258] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.705010 9957 ts_tablet_manager.cc:1875] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.705060 10336 ts_tablet_manager.cc:1875] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.706540 10335 tablet_replica.cc:330] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:13.706719 10335 raft_consensus.cc:2229] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.706902 10335 raft_consensus.cc:2258] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.707450 10335 ts_tablet_manager.cc:1875] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.707808 9767 raft_consensus.cc:1203] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.709704 10335 ts_tablet_manager.cc:1888] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.709784 10335 log.cc:1196] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c86d285233c84f628d2ce4c531e3aef4
I20240503 07:59:13.710011 10335 ts_tablet_manager.cc:1909] T c86d285233c84f628d2ce4c531e3aef4 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:13.710076 10336 ts_tablet_manager.cc:1888] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.710245 10336 log.cc:1196] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c86d285233c84f628d2ce4c531e3aef4
I20240503 07:59:13.710554 10336 ts_tablet_manager.cc:1909] T c86d285233c84f628d2ce4c531e3aef4 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:13.710603 6744 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 20d8ad35129042a093035013305e42b5
I20240503 07:59:13.710981 6744 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 4 replicas of tablet 5cbcf7ab42a1463582e96927d7f16b9a
I20240503 07:59:13.711186 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet c86d285233c84f628d2ce4c531e3aef4 (table rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]) successfully deleted
I20240503 07:59:13.711637 9353 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cbcf7ab42a1463582e96927d7f16b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:13.711642 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet 20d8ad35129042a093035013305e42b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:13.711836 10512 tablet_replica.cc:330] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:13.711849 10670 tablet_replica.cc:330] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:13.711863 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet 20d8ad35129042a093035013305e42b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:13.711970 10670 raft_consensus.cc:2229] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.711967 10512 raft_consensus.cc:2229] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.712072 10670 raft_consensus.cc:2258] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.712150 10512 raft_consensus.cc:2258] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.712426 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cbcf7ab42a1463582e96927d7f16b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:13.712659 10670 ts_tablet_manager.cc:1875] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.712718 10512 ts_tablet_manager.cc:1875] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.714654 7130 tablet_service.cc:1514] Processing DeleteTablet for tablet 20d8ad35129042a093035013305e42b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:13.714694 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cbcf7ab42a1463582e96927d7f16b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:13.714799 10512 ts_tablet_manager.cc:1888] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.714862 10512 log.cc:1196] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/20d8ad35129042a093035013305e42b5
I20240503 07:59:13.715085 10512 ts_tablet_manager.cc:1909] T 20d8ad35129042a093035013305e42b5 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:13.715750 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 20d8ad35129042a093035013305e42b5 (table rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]) successfully deleted
I20240503 07:59:13.716967 9957 ts_tablet_manager.cc:1888] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.717056 9957 log.cc:1196] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c86d285233c84f628d2ce4c531e3aef4
I20240503 07:59:13.717288 9957 ts_tablet_manager.cc:1909] T c86d285233c84f628d2ce4c531e3aef4 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:13.717927 9957 tablet_replica.cc:330] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:13.718075 9957 raft_consensus.cc:2229] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.718075 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet c86d285233c84f628d2ce4c531e3aef4 (table rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]) successfully deleted
I20240503 07:59:13.718204 9957 raft_consensus.cc:2258] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.719188 10670 ts_tablet_manager.cc:1888] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.719516 10670 log.cc:1196] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5cbcf7ab42a1463582e96927d7f16b9a
I20240503 07:59:13.719813 10670 ts_tablet_manager.cc:1909] T 5cbcf7ab42a1463582e96927d7f16b9a P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:13.720489 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 5cbcf7ab42a1463582e96927d7f16b9a (table rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]) successfully deleted
I20240503 07:59:13.723284 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cbcf7ab42a1463582e96927d7f16b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:13.723513 10336 tablet_replica.cc:330] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:13.723587 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet c86d285233c84f628d2ce4c531e3aef4 (table rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]) successfully deleted
I20240503 07:59:13.723742 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:13.723837 10336 raft_consensus.cc:2229] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.724061 10335 tablet_replica.cc:330] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:13.724231 10336 raft_consensus.cc:2258] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.724273 10335 raft_consensus.cc:2229] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.724511 10335 raft_consensus.cc:2258] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.725006 10336 ts_tablet_manager.cc:1875] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.725190 10335 ts_tablet_manager.cc:1875] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.727594 9636 raft_consensus.cc:1203] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.727746 10335 ts_tablet_manager.cc:1888] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.727844 10335 log.cc:1196] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/20d8ad35129042a093035013305e42b5
I20240503 07:59:13.729557 10335 ts_tablet_manager.cc:1909] T 20d8ad35129042a093035013305e42b5 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:13.730123 9957 ts_tablet_manager.cc:1875] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.731734 10336 ts_tablet_manager.cc:1888] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.732316 9353 tablet_service.cc:1467] Processing CreateTablet for tablet cf2beba18ec6449c90c50e200ff40d68 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=476bb7319d81414d9f735392a591b6a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.732347 6868 tablet_service.cc:1467] Processing CreateTablet for tablet cf2beba18ec6449c90c50e200ff40d68 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=476bb7319d81414d9f735392a591b6a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.732618 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2beba18ec6449c90c50e200ff40d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.732841 9616 tablet_service.cc:1467] Processing CreateTablet for tablet cf2beba18ec6449c90c50e200ff40d68 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=476bb7319d81414d9f735392a591b6a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.733150 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2beba18ec6449c90c50e200ff40d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.735289 9485 tablet_service.cc:1467] Processing CreateTablet for tablet bd6e2fa89d954521b69d097afcb4715b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=476bb7319d81414d9f735392a591b6a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.735563 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6e2fa89d954521b69d097afcb4715b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.735545 9615 tablet_service.cc:1467] Processing CreateTablet for tablet bd6e2fa89d954521b69d097afcb4715b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=476bb7319d81414d9f735392a591b6a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.735831 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6e2fa89d954521b69d097afcb4715b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.738197 9614 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2312d0a7934056bf438a78e6641226 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=476bb7319d81414d9f735392a591b6a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.738500 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2312d0a7934056bf438a78e6641226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.738649 10795 tablet_bootstrap.cc:492] T cf2beba18ec6449c90c50e200ff40d68 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:13.739713 10795 tablet_bootstrap.cc:654] T cf2beba18ec6449c90c50e200ff40d68 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.740705 10799 tablet_bootstrap.cc:492] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:13.740888 10847 tablet_bootstrap.cc:492] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:13.741672 10799 tablet_bootstrap.cc:654] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.742661 10795 tablet_bootstrap.cc:492] T cf2beba18ec6449c90c50e200ff40d68 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:13.742758 10795 ts_tablet_manager.cc:1367] T cf2beba18ec6449c90c50e200ff40d68 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:13.742708 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2312d0a7934056bf438a78e6641226 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=476bb7319d81414d9f735392a591b6a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.743008 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2312d0a7934056bf438a78e6641226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.743101 10335 tablet_replica.cc:330] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:13.743257 10335 raft_consensus.cc:2229] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.743253 10795 raft_consensus.cc:348] T cf2beba18ec6449c90c50e200ff40d68 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:13.743376 10795 raft_consensus.cc:374] T cf2beba18ec6449c90c50e200ff40d68 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.743407 10795 raft_consensus.cc:729] T cf2beba18ec6449c90c50e200ff40d68 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.743420 10335 raft_consensus.cc:2258] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.743538 10795 consensus_queue.cc:260] T cf2beba18ec6449c90c50e200ff40d68 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:13.743832 10795 ts_tablet_manager.cc:1398] T cf2beba18ec6449c90c50e200ff40d68 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:13.743944 10335 ts_tablet_manager.cc:1875] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.741878 10847 tablet_bootstrap.cc:654] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.746119 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 20d8ad35129042a093035013305e42b5 (table rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]) successfully deleted
I20240503 07:59:13.746583 6998 tablet_service.cc:1467] Processing CreateTablet for tablet bd6e2fa89d954521b69d097afcb4715b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=476bb7319d81414d9f735392a591b6a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.746851 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6e2fa89d954521b69d097afcb4715b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.747232 10847 tablet_bootstrap.cc:492] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:13.747335 10847 ts_tablet_manager.cc:1367] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:13.747890 10847 raft_consensus.cc:348] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:13.748014 10847 raft_consensus.cc:374] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.748047 10847 raft_consensus.cc:729] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.748175 10847 consensus_queue.cc:260] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:13.748447 10847 ts_tablet_manager.cc:1398] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.748636 10847 tablet_bootstrap.cc:492] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:13.749588 10847 tablet_bootstrap.cc:654] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.750865 10335 ts_tablet_manager.cc:1888] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.750854 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2312d0a7934056bf438a78e6641226 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=476bb7319d81414d9f735392a591b6a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.750944 10335 log.cc:1196] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5cbcf7ab42a1463582e96927d7f16b9a
I20240503 07:59:13.751173 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2312d0a7934056bf438a78e6641226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.751389 10335 ts_tablet_manager.cc:1909] T 5cbcf7ab42a1463582e96927d7f16b9a P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:13.751915 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2beba18ec6449c90c50e200ff40d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.753070 10847 tablet_bootstrap.cc:492] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:13.753163 10847 ts_tablet_manager.cc:1367] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:13.753224 9957 ts_tablet_manager.cc:1888] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.753291 9957 log.cc:1196] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/20d8ad35129042a093035013305e42b5
I20240503 07:59:13.753544 9957 ts_tablet_manager.cc:1909] T 20d8ad35129042a093035013305e42b5 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:13.753679 10847 raft_consensus.cc:348] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.753809 10847 raft_consensus.cc:374] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.753841 10847 raft_consensus.cc:729] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.753962 10847 consensus_queue.cc:260] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.754171 9957 tablet_replica.cc:330] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:13.754264 10847 ts_tablet_manager.cc:1398] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.754309 9957 raft_consensus.cc:2229] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.754441 10847 tablet_bootstrap.cc:492] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:13.754590 9957 raft_consensus.cc:2258] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.755121 9957 ts_tablet_manager.cc:1875] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.755545 10847 tablet_bootstrap.cc:654] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.756601 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 20d8ad35129042a093035013305e42b5 (table rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]) successfully deleted
I20240503 07:59:13.756693 10852 tablet_bootstrap.cc:492] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:13.757769 9957 ts_tablet_manager.cc:1888] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.757923 9957 log.cc:1196] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5cbcf7ab42a1463582e96927d7f16b9a
I20240503 07:59:13.758055 10852 tablet_bootstrap.cc:654] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.758237 9957 ts_tablet_manager.cc:1909] T 5cbcf7ab42a1463582e96927d7f16b9a P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:13.759059 10336 log.cc:1196] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5cbcf7ab42a1463582e96927d7f16b9a
I20240503 07:59:13.759431 10336 ts_tablet_manager.cc:1909] T 5cbcf7ab42a1463582e96927d7f16b9a P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:13.760398 10849 tablet_bootstrap.cc:492] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:13.760772 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 5cbcf7ab42a1463582e96927d7f16b9a (table rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]) successfully deleted
I20240503 07:59:13.760788 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 5cbcf7ab42a1463582e96927d7f16b9a (table rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]) successfully deleted
I20240503 07:59:13.761608 10853 tablet_bootstrap.cc:492] T cf2beba18ec6449c90c50e200ff40d68 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:13.761654 10847 tablet_bootstrap.cc:492] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:13.761744 10847 ts_tablet_manager.cc:1367] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:13.761783 10849 tablet_bootstrap.cc:654] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.762269 10847 raft_consensus.cc:348] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:13.762423 10847 raft_consensus.cc:374] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.762461 10847 raft_consensus.cc:729] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.762586 10847 consensus_queue.cc:260] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:13.762872 10847 ts_tablet_manager.cc:1398] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.761330 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 5cbcf7ab42a1463582e96927d7f16b9a (table rebalance_test_table_10 [id=729e1f3549764c26aff60e7a36618095]) successfully deleted
I20240503 07:59:13.762887 10849 tablet_bootstrap.cc:492] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:13.762980 10849 ts_tablet_manager.cc:1367] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:13.763240 10799 tablet_bootstrap.cc:492] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:13.763348 10799 ts_tablet_manager.cc:1367] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240503 07:59:13.763501 10849 raft_consensus.cc:348] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:13.763612 10849 raft_consensus.cc:374] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.763643 10849 raft_consensus.cc:729] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.763767 10849 consensus_queue.cc:260] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:13.763825 10799 raft_consensus.cc:348] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.763931 10799 raft_consensus.cc:374] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.763962 10799 raft_consensus.cc:729] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.764053 10849 ts_tablet_manager.cc:1398] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.764081 10799 consensus_queue.cc:260] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.764245 10849 tablet_bootstrap.cc:492] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:13.764351 10799 ts_tablet_manager.cc:1398] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.765218 10849 tablet_bootstrap.cc:654] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.765715 10853 tablet_bootstrap.cc:654] T cf2beba18ec6449c90c50e200ff40d68 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.770761 10853 tablet_bootstrap.cc:492] T cf2beba18ec6449c90c50e200ff40d68 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:13.770956 10853 ts_tablet_manager.cc:1367] T cf2beba18ec6449c90c50e200ff40d68 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240503 07:59:13.771600 10852 tablet_bootstrap.cc:492] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:13.771760 10852 ts_tablet_manager.cc:1367] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:13.771528 10853 raft_consensus.cc:348] T cf2beba18ec6449c90c50e200ff40d68 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:13.771941 10853 raft_consensus.cc:374] T cf2beba18ec6449c90c50e200ff40d68 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.772022 10853 raft_consensus.cc:729] T cf2beba18ec6449c90c50e200ff40d68 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.772219 10853 consensus_queue.cc:260] T cf2beba18ec6449c90c50e200ff40d68 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:13.772361 10852 raft_consensus.cc:348] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:13.772599 10852 raft_consensus.cc:374] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.772619 10853 ts_tablet_manager.cc:1398] T cf2beba18ec6449c90c50e200ff40d68 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:13.772692 10852 raft_consensus.cc:729] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.772867 10852 consensus_queue.cc:260] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:13.773171 10852 ts_tablet_manager.cc:1398] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.774014 10849 tablet_bootstrap.cc:492] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:13.774161 10849 ts_tablet_manager.cc:1367] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:13.774668 10849 raft_consensus.cc:348] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.774786 10849 raft_consensus.cc:374] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.774819 10849 raft_consensus.cc:729] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.774933 10849 consensus_queue.cc:260] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:13.775218 10849 ts_tablet_manager.cc:1398] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.783567 10694 raft_consensus.cc:482] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:13.783690 10694 raft_consensus.cc:504] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.784101 10694 leader_election.cc:288] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:13.784567 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5f760fd6b64ab796c6d8355fa3616c" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:13.784561 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5f760fd6b64ab796c6d8355fa3616c" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:13.784708 9504 raft_consensus.cc:2454] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:13.784711 9767 raft_consensus.cc:2454] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:13.785014 9310 leader_election.cc:302] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:13.785198 10694 raft_consensus.cc:2781] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:13.785261 10694 raft_consensus.cc:482] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:13.785306 10694 raft_consensus.cc:3037] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.786280 10694 raft_consensus.cc:504] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.786721 10694 leader_election.cc:288] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:13.787248 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5f760fd6b64ab796c6d8355fa3616c" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:13.787254 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5f760fd6b64ab796c6d8355fa3616c" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:13.787384 9767 raft_consensus.cc:3037] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.787397 9504 raft_consensus.cc:3037] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.788211 9504 raft_consensus.cc:2454] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:13.788394 9767 raft_consensus.cc:2454] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:13.788683 9310 leader_election.cc:302] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:13.788872 10694 raft_consensus.cc:2781] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:13.788960 10694 raft_consensus.cc:686] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:13.789105 10694 consensus_queue.cc:237] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:13.791131 6741 catalog_manager.cc:5484] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:13.809775 9614 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:13.810236 7129 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:13.823345 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:13.827316 6868 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:13.828534 7261 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:13.833029 9485 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:13.834188 9374 raft_consensus.cc:1203] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.838213 9746 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:13.852468 10310 raft_consensus.cc:482] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:13.852607 10310 raft_consensus.cc:504] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:13.853018 10310 leader_election.cc:288] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:13.853294 6998 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:13.855265 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2312d0a7934056bf438a78e6641226" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:13.855463 9636 raft_consensus.cc:2454] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:13.856783 6953 leader_election.cc:302] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:13.857831 10310 raft_consensus.cc:2781] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:13.857910 10310 raft_consensus.cc:482] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:13.857944 10310 raft_consensus.cc:3037] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.859124 10310 raft_consensus.cc:504] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:13.859547 10310 leader_election.cc:288] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:13.860361 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2312d0a7934056bf438a78e6641226" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:13.860509 9636 raft_consensus.cc:3037] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.860673 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2312d0a7934056bf438a78e6641226" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:13.860810 7278 raft_consensus.cc:3037] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.860849 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2312d0a7934056bf438a78e6641226" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:13.861686 9636 raft_consensus.cc:2454] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:13.861773 7278 raft_consensus.cc:2454] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:13.862123 6956 leader_election.cc:302] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:13.862425 10310 raft_consensus.cc:2781] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:13.862516 10310 raft_consensus.cc:686] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:13.862648 10310 consensus_queue.cc:237] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:13.865072 6741 catalog_manager.cc:5484] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:13.895337 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:13.901093 9504 raft_consensus.cc:1261] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:13.901118 9767 raft_consensus.cc:1261] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:13.901551 10694 consensus_queue.cc:1035] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.901861 10694 consensus_queue.cc:1035] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.904948 10886 tablet.cc:1722] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:13.905259 6741 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 7d5f760fd6b64ab796c6d8355fa3616c (table rebalancer_dynamic_table [id=7d6b27aaf354421c8436d0a1436d673c]). Expected version 1 got 0
I20240503 07:59:13.905326 10886 tablet.cc:1620] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:13.906801 6741 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 7d5f760fd6b64ab796c6d8355fa3616c (table rebalancer_dynamic_table [id=7d6b27aaf354421c8436d0a1436d673c]). Expected version 1 got 0
I20240503 07:59:13.907338 6745 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 7d5f760fd6b64ab796c6d8355fa3616c (table rebalancer_dynamic_table [id=7d6b27aaf354421c8436d0a1436d673c]). Expected version 1 got 0
I20240503 07:59:13.907975 10887 tablet.cc:1722] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:13.908385 10887 tablet.cc:1620] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:13.909229 10888 tablet.cc:1722] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:13.909605 10888 tablet.cc:1620] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:13.909637 6741 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 7d5f760fd6b64ab796c6d8355fa3616c (table rebalancer_dynamic_table [id=7d6b27aaf354421c8436d0a1436d673c]). Expected version 1 got 0
I20240503 07:59:13.911808 10889 ts_tablet_manager.cc:902] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:13.912067 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=7d6b27aaf354421c8436d0a1436d673c] alter complete (version 1)
I20240503 07:59:13.912530 10889 tablet_copy_client.cc:308] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:13.918737 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 5365969434f142b8a5757a9a73bc109a from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:48617)
I20240503 07:59:13.918881 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 5365969434f142b8a5757a9a73bc109a from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:48617: session id = cb73994515af4880b05383e78368bb4a-5365969434f142b8a5757a9a73bc109a
I20240503 07:59:13.919833 7171 tablet_copy_source_session.cc:205] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.920833 10889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5365969434f142b8a5757a9a73bc109a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.923332 10889 tablet_copy_client.cc:783] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.923544 10889 tablet_copy_client.cc:647] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.924758 10889 tablet_copy_client.cc:523] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.925940 10889 tablet_bootstrap.cc:492] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:13.951336 10889 tablet_bootstrap.cc:492] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.951701 10889 tablet_bootstrap.cc:492] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:13.951999 10889 ts_tablet_manager.cc:1367] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.026s user 0.019s sys 0.005s
I20240503 07:59:13.952565 10889 raft_consensus.cc:348] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:13.952716 10889 raft_consensus.cc:729] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.952875 10889 consensus_queue.cc:260] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:13.953320 10889 ts_tablet_manager.cc:1398] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:59:13.953946 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-5365969434f142b8a5757a9a73bc109a received from {username='slave'} at 127.0.99.134:48617
I20240503 07:59:13.954041 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session cb73994515af4880b05383e78368bb4a-5365969434f142b8a5757a9a73bc109a on tablet 5365969434f142b8a5757a9a73bc109a with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:13.986598 10925 ts_tablet_manager.cc:902] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:13.987190 10925 tablet_copy_client.cc:308] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:13.987769 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 16c06ab5a3e54ed2910afa50069810cc from peer a372e98a670f49e9adca46ab5c836c4a ({username='slave'} at 127.0.99.136:53081)
I20240503 07:59:13.987897 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 16c06ab5a3e54ed2910afa50069810cc from peer a372e98a670f49e9adca46ab5c836c4a at {username='slave'} at 127.0.99.136:53081: session id = a372e98a670f49e9adca46ab5c836c4a-16c06ab5a3e54ed2910afa50069810cc
I20240503 07:59:13.989004 7171 tablet_copy_source_session.cc:205] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.990315 10925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c06ab5a3e54ed2910afa50069810cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.992976 10925 tablet_copy_client.cc:783] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.993249 10925 tablet_copy_client.cc:647] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.994989 10925 tablet_copy_client.cc:523] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.996366 10925 tablet_bootstrap.cc:492] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:14.021139 10925 tablet_bootstrap.cc:492] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:14.021484 10925 tablet_bootstrap.cc:492] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a: Bootstrap complete.
I20240503 07:59:14.021678 10925 ts_tablet_manager.cc:1367] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240503 07:59:14.021893 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:14.022192 10925 raft_consensus.cc:348] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:14.022313 10925 raft_consensus.cc:729] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.022506 10925 consensus_queue.cc:260] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:14.022904 10925 ts_tablet_manager.cc:1398] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:14.025426 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session a372e98a670f49e9adca46ab5c836c4a-16c06ab5a3e54ed2910afa50069810cc received from {username='slave'} at 127.0.99.136:53081
I20240503 07:59:14.025550 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session a372e98a670f49e9adca46ab5c836c4a-16c06ab5a3e54ed2910afa50069810cc on tablet 16c06ab5a3e54ed2910afa50069810cc with peer a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:14.029323 10886 tablet.cc:1722] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:14.029678 10886 tablet.cc:1620] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:14.029716 10888 tablet.cc:1722] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:14.029899 10887 tablet.cc:1722] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:14.030030 10888 tablet.cc:1620] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:14.030191 10887 tablet.cc:1620] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:14.036454 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=7d6b27aaf354421c8436d0a1436d673c] alter complete (version 2)
I20240503 07:59:14.050276 10801 raft_consensus.cc:482] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:14.050449 10801 raft_consensus.cc:504] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:14.050951 10801 leader_election.cc:288] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:14.051333 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2beba18ec6449c90c50e200ff40d68" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:14.051403 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2beba18ec6449c90c50e200ff40d68" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:14.051493 6888 raft_consensus.cc:2454] T cf2beba18ec6449c90c50e200ff40d68 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:14.051585 9374 raft_consensus.cc:2454] T cf2beba18ec6449c90c50e200ff40d68 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:14.051807 9569 leader_election.cc:302] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:14.051965 10801 raft_consensus.cc:2781] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:14.052027 10801 raft_consensus.cc:482] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:14.052070 10801 raft_consensus.cc:3037] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.053097 10801 raft_consensus.cc:504] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:14.053463 10801 leader_election.cc:288] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:14.053889 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2beba18ec6449c90c50e200ff40d68" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:14.054014 6888 raft_consensus.cc:3037] T cf2beba18ec6449c90c50e200ff40d68 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.054015 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2beba18ec6449c90c50e200ff40d68" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:14.054177 9374 raft_consensus.cc:3037] T cf2beba18ec6449c90c50e200ff40d68 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.054991 6888 raft_consensus.cc:2454] T cf2beba18ec6449c90c50e200ff40d68 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:14.055035 9374 raft_consensus.cc:2454] T cf2beba18ec6449c90c50e200ff40d68 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:14.055305 9569 leader_election.cc:302] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:14.055457 10801 raft_consensus.cc:2781] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:14.055555 10801 raft_consensus.cc:686] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:14.055728 10801 consensus_queue.cc:237] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:14.057547 6738 catalog_manager.cc:5484] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:14.140076 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:14.145824 10886 tablet.cc:1722] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:14.146471 10888 tablet.cc:1722] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:14.146772 10887 tablet.cc:1722] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:14.150841 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=7d6b27aaf354421c8436d0a1436d673c] alter complete (version 3)
I20240503 07:59:14.161752 10379 raft_consensus.cc:482] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:14.161938 10379 raft_consensus.cc:504] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:14.162629 10379 leader_election.cc:288] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:14.163369 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6e2fa89d954521b69d097afcb4715b" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:14.163630 7018 raft_consensus.cc:2454] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:14.164003 9440 leader_election.cc:302] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:14.164216 10379 raft_consensus.cc:2781] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:14.164292 10379 raft_consensus.cc:482] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:14.164325 10379 raft_consensus.cc:3037] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.165424 10379 raft_consensus.cc:504] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:14.165823 10379 leader_election.cc:288] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:14.166239 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6e2fa89d954521b69d097afcb4715b" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:14.166424 7018 raft_consensus.cc:3037] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.166728 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6e2fa89d954521b69d097afcb4715b" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:14.166770 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6e2fa89d954521b69d097afcb4715b" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:14.166867 9636 raft_consensus.cc:2454] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:14.167497 7018 raft_consensus.cc:2454] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:14.167872 9440 leader_election.cc:302] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:14.168305 10379 raft_consensus.cc:2781] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:14.168476 10379 raft_consensus.cc:686] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:14.168622 10379 consensus_queue.cc:237] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:14.170611 6746 catalog_manager.cc:5484] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:14.200186 7129 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:14.201839 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:14.206291 9746 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:14.207464 6868 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:14.209654 7261 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:14.211041 6998 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:14.213021 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:14.227332 6746 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 07:59:14.227535 6746 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 07:59:14.232327 6746 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 4 replicas of tablet 85bf549243424602b6615773cf8bee33
I20240503 07:59:14.232724 6746 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet e32f28262dfb4c738bf8e939902ef181
I20240503 07:59:14.232920 6746 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 4 replicas of tablet 96c14f4b2a054ea98bde16bc8d4422ac
I20240503 07:59:14.233682 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet 85bf549243424602b6615773cf8bee33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:14.233894 9746 tablet_service.cc:1514] Processing DeleteTablet for tablet 85bf549243424602b6615773cf8bee33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:14.233983 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c14f4b2a054ea98bde16bc8d4422ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:14.234702 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet e32f28262dfb4c738bf8e939902ef181 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:14.234741 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c14f4b2a054ea98bde16bc8d4422ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:14.234941 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet 85bf549243424602b6615773cf8bee33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:14.234962 9614 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c14f4b2a054ea98bde16bc8d4422ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:14.234982 9957 tablet_replica.cc:330] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:14.235168 9957 raft_consensus.cc:2229] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.235332 9957 raft_consensus.cc:2258] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.235936 9957 ts_tablet_manager.cc:1875] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.238999 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 85bf549243424602b6615773cf8bee33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:14.239162 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet e32f28262dfb4c738bf8e939902ef181 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:14.239280 10336 tablet_replica.cc:330] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:14.239403 10336 raft_consensus.cc:2229] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.239606 7130 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c14f4b2a054ea98bde16bc8d4422ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:14.239667 10336 raft_consensus.cc:2258] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.240288 10336 ts_tablet_manager.cc:1875] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.240398 9957 ts_tablet_manager.cc:1888] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.240476 9957 log.cc:1196] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e32f28262dfb4c738bf8e939902ef181
I20240503 07:59:14.241078 9957 ts_tablet_manager.cc:1909] T e32f28262dfb4c738bf8e939902ef181 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:14.242480 10335 tablet_replica.cc:330] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:14.242455 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet e32f28262dfb4c738bf8e939902ef181 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:14.242539 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet e32f28262dfb4c738bf8e939902ef181 (table rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]) successfully deleted
I20240503 07:59:14.242637 10335 raft_consensus.cc:2229] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.242862 10336 ts_tablet_manager.cc:1888] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.242908 10335 raft_consensus.cc:2258] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.243213 10336 log.cc:1196] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/85bf549243424602b6615773cf8bee33
I20240503 07:59:14.243093 10948 tablet_replica.cc:330] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:14.243407 10948 raft_consensus.cc:2229] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.243446 10335 ts_tablet_manager.cc:1875] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.243538 10948 raft_consensus.cc:2258] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.243670 10336 ts_tablet_manager.cc:1909] T 85bf549243424602b6615773cf8bee33 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:14.244290 10336 tablet_replica.cc:330] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:14.244469 10336 raft_consensus.cc:2229] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.244633 10336 raft_consensus.cc:2258] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.245138 10336 ts_tablet_manager.cc:1875] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.245146 9957 tablet_replica.cc:330] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:14.245285 9957 raft_consensus.cc:2229] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.245388 10949 tablet_replica.cc:330] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:14.245461 9957 raft_consensus.cc:2258] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.245534 10949 raft_consensus.cc:2229] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.245666 10949 raft_consensus.cc:2258] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.245800 10335 ts_tablet_manager.cc:1888] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.245882 10335 log.cc:1196] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/96c14f4b2a054ea98bde16bc8d4422ac
I20240503 07:59:14.245971 9957 ts_tablet_manager.cc:1875] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.246134 10335 ts_tablet_manager.cc:1909] T 96c14f4b2a054ea98bde16bc8d4422ac P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:14.246153 10949 ts_tablet_manager.cc:1875] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.246536 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 85bf549243424602b6615773cf8bee33 (table rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]) successfully deleted
I20240503 07:59:14.246914 10948 ts_tablet_manager.cc:1875] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.247891 10336 ts_tablet_manager.cc:1888] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.248068 10336 log.cc:1196] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e32f28262dfb4c738bf8e939902ef181
I20240503 07:59:14.248427 10336 ts_tablet_manager.cc:1909] T e32f28262dfb4c738bf8e939902ef181 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:14.249146 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet e32f28262dfb4c738bf8e939902ef181 (table rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]) successfully deleted
I20240503 07:59:14.251188 10335 tablet_replica.cc:330] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:14.251586 10335 raft_consensus.cc:2229] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.251839 10335 raft_consensus.cc:2258] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.252429 10335 ts_tablet_manager.cc:1875] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.254868 10335 ts_tablet_manager.cc:1888] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.255084 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 96c14f4b2a054ea98bde16bc8d4422ac (table rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]) successfully deleted
I20240503 07:59:14.255501 10947 tablet_replica.cc:330] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:14.255882 6746 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:14.255957 10949 ts_tablet_manager.cc:1888] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.256022 6746 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:14.256036 10949 log.cc:1196] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/96c14f4b2a054ea98bde16bc8d4422ac
I20240503 07:59:14.256285 10949 ts_tablet_manager.cc:1909] T 96c14f4b2a054ea98bde16bc8d4422ac P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:14.256554 10947 raft_consensus.cc:2229] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.256985 10335 log.cc:1196] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e32f28262dfb4c738bf8e939902ef181
I20240503 07:59:14.257112 10947 raft_consensus.cc:2258] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.257400 10335 ts_tablet_manager.cc:1909] T e32f28262dfb4c738bf8e939902ef181 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:14.257647 10947 ts_tablet_manager.cc:1875] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.258286 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 96c14f4b2a054ea98bde16bc8d4422ac (table rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]) successfully deleted
I20240503 07:59:14.258898 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet e32f28262dfb4c738bf8e939902ef181 (table rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]) successfully deleted
I20240503 07:59:14.259578 10948 ts_tablet_manager.cc:1888] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.259646 10948 log.cc:1196] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/85bf549243424602b6615773cf8bee33
I20240503 07:59:14.259902 10948 ts_tablet_manager.cc:1909] T 85bf549243424602b6615773cf8bee33 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:14.261629 10947 ts_tablet_manager.cc:1888] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.261685 10947 log.cc:1196] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/85bf549243424602b6615773cf8bee33
I20240503 07:59:14.261858 10947 ts_tablet_manager.cc:1909] T 85bf549243424602b6615773cf8bee33 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:14.261909 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 85bf549243424602b6615773cf8bee33 (table rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]) successfully deleted
I20240503 07:59:14.262243 6746 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 7d5f760fd6b64ab796c6d8355fa3616c
I20240503 07:59:14.263084 9353 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5f760fd6b64ab796c6d8355fa3616c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:14.263195 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5f760fd6b64ab796c6d8355fa3616c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:14.263929 9746 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5f760fd6b64ab796c6d8355fa3616c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:14.266778 9957 ts_tablet_manager.cc:1888] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.266872 9957 log.cc:1196] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/96c14f4b2a054ea98bde16bc8d4422ac
I20240503 07:59:14.267120 9957 ts_tablet_manager.cc:1909] T 96c14f4b2a054ea98bde16bc8d4422ac P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:14.268065 9485 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:14.268556 10948 tablet_replica.cc:330] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:14.268721 10948 raft_consensus.cc:2229] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.268846 10948 raft_consensus.cc:2258] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.269568 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 96c14f4b2a054ea98bde16bc8d4422ac (table rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]) successfully deleted
I20240503 07:59:14.270900 10947 tablet_replica.cc:330] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:14.271059 10947 raft_consensus.cc:2229] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.271224 10947 raft_consensus.cc:2258] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.272745 9957 tablet_replica.cc:330] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:14.272903 9957 raft_consensus.cc:2229] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.273046 9957 raft_consensus.cc:2258] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.273556 9957 ts_tablet_manager.cc:1875] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.275974 10948 ts_tablet_manager.cc:1875] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.277379 10950 tablet_replica.cc:330] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:14.277606 10950 raft_consensus.cc:2229] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.277958 10950 raft_consensus.cc:2258] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.278666 10950 ts_tablet_manager.cc:1875] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.281705 10951 tablet_replica.cc:330] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:14.281685 10950 ts_tablet_manager.cc:1888] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.281777 10950 log.cc:1196] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7d5f760fd6b64ab796c6d8355fa3616c
I20240503 07:59:14.282009 10951 raft_consensus.cc:2229] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.282073 10950 ts_tablet_manager.cc:1909] T 7d5f760fd6b64ab796c6d8355fa3616c P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:14.282855 10947 ts_tablet_manager.cc:1875] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.282974 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 7d5f760fd6b64ab796c6d8355fa3616c (table rebalancer_dynamic_table_new_name [id=7d6b27aaf354421c8436d0a1436d673c]) successfully deleted
I20240503 07:59:14.283056 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 85bf549243424602b6615773cf8bee33 (table rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]) successfully deleted
I20240503 07:59:14.283296 10951 raft_consensus.cc:2258] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.285481 10951 ts_tablet_manager.cc:1875] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.286728 10948 ts_tablet_manager.cc:1888] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.286892 10948 log.cc:1196] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/7d5f760fd6b64ab796c6d8355fa3616c
I20240503 07:59:14.287186 10948 ts_tablet_manager.cc:1909] T 7d5f760fd6b64ab796c6d8355fa3616c P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:14.288728 9957 ts_tablet_manager.cc:1888] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.288827 9957 log.cc:1196] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/85bf549243424602b6615773cf8bee33
I20240503 07:59:14.289060 9957 ts_tablet_manager.cc:1909] T 85bf549243424602b6615773cf8bee33 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:14.289681 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 7d5f760fd6b64ab796c6d8355fa3616c (table rebalancer_dynamic_table_new_name [id=7d6b27aaf354421c8436d0a1436d673c]) successfully deleted
I20240503 07:59:14.289719 10947 ts_tablet_manager.cc:1888] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.289796 10947 log.cc:1196] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/96c14f4b2a054ea98bde16bc8d4422ac
I20240503 07:59:14.289856 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 85bf549243424602b6615773cf8bee33 (table rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]) successfully deleted
I20240503 07:59:14.290031 10947 ts_tablet_manager.cc:1909] T 96c14f4b2a054ea98bde16bc8d4422ac P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:14.291960 10951 ts_tablet_manager.cc:1888] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.292109 10951 log.cc:1196] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/7d5f760fd6b64ab796c6d8355fa3616c
I20240503 07:59:14.292461 10951 ts_tablet_manager.cc:1909] T 7d5f760fd6b64ab796c6d8355fa3616c P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:14.293296 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 7d5f760fd6b64ab796c6d8355fa3616c (table rebalancer_dynamic_table_new_name [id=7d6b27aaf354421c8436d0a1436d673c]) successfully deleted
I20240503 07:59:14.294579 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 96c14f4b2a054ea98bde16bc8d4422ac (table rebalance_test_table_12 [id=552b9212b8a6475b947a9931f8e4bb4f]) successfully deleted
I20240503 07:59:14.321797 7019 consensus_queue.cc:237] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:14.323297 9635 raft_consensus.cc:1261] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.323359 7278 raft_consensus.cc:1261] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.323753 10310 consensus_queue.cc:1035] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.324191 10310 consensus_queue.cc:1035] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.325835 9151 raft_consensus.cc:2932] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:14.326520 9635 raft_consensus.cc:2932] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:14.328541 7278 raft_consensus.cc:2932] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:14.329090 6746 catalog_manager.cc:5484] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:14.333734 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c76213f3ec451084ae7b0551cd7b48 with delete_type TABLET_DATA_TOMBSTONED (TS 00a3f8e159c64327832d79fb32cdcca8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:14.334031 10335 tablet_replica.cc:330] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:14.334218 10335 raft_consensus.cc:2229] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.334406 10335 raft_consensus.cc:2258] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.335065 10335 ts_tablet_manager.cc:1875] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.337466 10335 ts_tablet_manager.cc:1888] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:14.337527 10335 log.cc:1196] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/31c76213f3ec451084ae7b0551cd7b48
I20240503 07:59:14.337992 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 31c76213f3ec451084ae7b0551cd7b48 (table rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]) successfully deleted
I20240503 07:59:14.356007 9151 consensus_queue.cc:1035] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.360000 9151 consensus_queue.cc:1035] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.366674 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:14.374040 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 70abff3c68be431d8141ba7bfc5866d2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ed7effdcb048462abc649fdf32202fa9]), partition=
I20240503 07:59:14.374723 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70abff3c68be431d8141ba7bfc5866d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.375121 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 70abff3c68be431d8141ba7bfc5866d2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ed7effdcb048462abc649fdf32202fa9]), partition=
I20240503 07:59:14.375384 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70abff3c68be431d8141ba7bfc5866d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.377291 10997 tablet_bootstrap.cc:492] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:14.377928 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 70abff3c68be431d8141ba7bfc5866d2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ed7effdcb048462abc649fdf32202fa9]), partition=
I20240503 07:59:14.378171 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70abff3c68be431d8141ba7bfc5866d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.378532 10998 tablet_bootstrap.cc:492] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:14.381028 10997 tablet_bootstrap.cc:654] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.381646 10998 tablet_bootstrap.cc:654] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.382544 10997 tablet_bootstrap.cc:492] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:14.382750 10997 ts_tablet_manager.cc:1367] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:14.383891 10997 raft_consensus.cc:348] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:14.384166 10997 raft_consensus.cc:374] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.384267 10997 raft_consensus.cc:729] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.384871 10997 consensus_queue.cc:260] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:14.385305 10997 ts_tablet_manager.cc:1398] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:14.386394 10998 tablet_bootstrap.cc:492] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:14.386544 10998 ts_tablet_manager.cc:1367] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:14.386572 10999 tablet_bootstrap.cc:492] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:14.387717 10999 tablet_bootstrap.cc:654] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.387712 10998 raft_consensus.cc:348] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:14.387812 9504 raft_consensus.cc:1203] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:14.387924 10998 raft_consensus.cc:374] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.387966 10998 raft_consensus.cc:729] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.388100 10998 consensus_queue.cc:260] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:14.389147 10998 ts_tablet_manager.cc:1398] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:14.405282 10999 tablet_bootstrap.cc:492] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:14.405494 10999 ts_tablet_manager.cc:1367] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 07:59:14.406078 10999 raft_consensus.cc:348] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:14.406317 10999 raft_consensus.cc:374] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.406450 10999 raft_consensus.cc:729] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.406659 10999 consensus_queue.cc:260] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:14.411618 10999 ts_tablet_manager.cc:1398] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:14.462134 10310 raft_consensus.cc:482] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:14.462289 10310 raft_consensus.cc:504] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:14.462759 10310 leader_election.cc:288] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:14.463197 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70abff3c68be431d8141ba7bfc5866d2" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:14.463266 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70abff3c68be431d8141ba7bfc5866d2" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:14.463380 9767 raft_consensus.cc:2454] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:14.463433 9504 raft_consensus.cc:2454] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:14.463820 6956 leader_election.cc:302] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:14.464069 10310 raft_consensus.cc:2781] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:14.464140 10310 raft_consensus.cc:482] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:14.464174 10310 raft_consensus.cc:3037] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.465404 10310 raft_consensus.cc:504] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:14.465822 10310 leader_election.cc:288] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:14.466267 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70abff3c68be431d8141ba7bfc5866d2" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:14.466262 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70abff3c68be431d8141ba7bfc5866d2" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:14.466435 9504 raft_consensus.cc:3037] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.466435 9767 raft_consensus.cc:3037] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.467536 9767 raft_consensus.cc:2454] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:14.467536 9504 raft_consensus.cc:2454] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:14.467923 6956 leader_election.cc:302] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:14.468120 10310 raft_consensus.cc:2781] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:14.468250 10310 raft_consensus.cc:686] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:14.468405 10310 consensus_queue.cc:237] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:14.470669 6742 catalog_manager.cc:5484] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:14.488232 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:14.489681 7129 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:14.496816 9353 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:14.502718 6868 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:14.506615 7260 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:14.510372 6999 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:14.520092 9746 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:14.531011 10801 consensus_queue.cc:1035] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.545496 9837 consensus_queue.cc:1035] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.553985 9485 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:14.562541 9635 raft_consensus.cc:3037] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.564450 10379 consensus_queue.cc:1035] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.572134 10379 consensus_queue.cc:1035] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.581223 9767 raft_consensus.cc:1203] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:14.586573 6742 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:14.592036 9504 raft_consensus.cc:1261] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:14.592515 9767 raft_consensus.cc:1261] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:14.592805 9151 consensus_queue.cc:1035] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.593130 9151 consensus_queue.cc:1035] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.596174 6742 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 70abff3c68be431d8141ba7bfc5866d2 (table rebalancer_dynamic_table [id=ed7effdcb048462abc649fdf32202fa9]). Expected version 1 got 0
I20240503 07:59:14.597226 11051 tablet.cc:1722] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:14.599062 11051 tablet.cc:1620] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: MemRowSet was empty: no flush needed.
I20240503 07:59:14.599659 6742 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 70abff3c68be431d8141ba7bfc5866d2 (table rebalancer_dynamic_table [id=ed7effdcb048462abc649fdf32202fa9]). Expected version 1 got 0
I20240503 07:59:14.601150 10887 tablet.cc:1722] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:14.601368 10888 tablet.cc:1722] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:14.601526 10887 tablet.cc:1620] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:14.601697 10888 tablet.cc:1620] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:14.606748 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=ed7effdcb048462abc649fdf32202fa9] alter complete (version 1)
I20240503 07:59:14.619858 7149 consensus_queue.cc:237] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:14.621455 7278 raft_consensus.cc:1261] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.621454 9504 raft_consensus.cc:1261] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.621944 11058 consensus_queue.cc:1035] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.622121 10716 consensus_queue.cc:1035] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.624231 10716 raft_consensus.cc:2932] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:14.624770 7278 raft_consensus.cc:2932] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:14.624931 9504 raft_consensus.cc:2932] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:14.628278 6741 catalog_manager.cc:5484] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:14.632458 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 5365969434f142b8a5757a9a73bc109a with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:14.632737 10336 tablet_replica.cc:330] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:14.632915 10336 raft_consensus.cc:2229] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.633132 10336 raft_consensus.cc:2258] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.633795 10336 ts_tablet_manager.cc:1875] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.636361 10336 ts_tablet_manager.cc:1888] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:14.636430 10336 log.cc:1196] T 5365969434f142b8a5757a9a73bc109a P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5365969434f142b8a5757a9a73bc109a
I20240503 07:59:14.637151 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 5365969434f142b8a5757a9a73bc109a (table rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]) successfully deleted
I20240503 07:59:14.712374 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:14.715807 6743 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:14.723675 11051 tablet.cc:1722] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:14.724092 11051 tablet.cc:1620] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: MemRowSet was empty: no flush needed.
I20240503 07:59:14.725956 9353 tablet_service.cc:1467] Processing CreateTablet for tablet 75f59b9fe4a54e7d8beabce7589800b1 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=914b54d652844204bb800dbd8a5be98a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.725994 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 75f59b9fe4a54e7d8beabce7589800b1 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=914b54d652844204bb800dbd8a5be98a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.726159 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 75f59b9fe4a54e7d8beabce7589800b1 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=914b54d652844204bb800dbd8a5be98a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.726306 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f59b9fe4a54e7d8beabce7589800b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.726336 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f59b9fe4a54e7d8beabce7589800b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.726475 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f59b9fe4a54e7d8beabce7589800b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.726605 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 517c0e477a8c40fbb84c4cd3bb6ecabd (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=914b54d652844204bb800dbd8a5be98a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.726857 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517c0e477a8c40fbb84c4cd3bb6ecabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.728843 10998 tablet_bootstrap.cc:492] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:14.729547 9353 tablet_service.cc:1467] Processing CreateTablet for tablet 517c0e477a8c40fbb84c4cd3bb6ecabd (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=914b54d652844204bb800dbd8a5be98a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.729818 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517c0e477a8c40fbb84c4cd3bb6ecabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.729856 10998 tablet_bootstrap.cc:654] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.730455 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 517c0e477a8c40fbb84c4cd3bb6ecabd (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=914b54d652844204bb800dbd8a5be98a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.730747 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517c0e477a8c40fbb84c4cd3bb6ecabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.731137 10887 tablet.cc:1722] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:14.731137 10888 tablet.cc:1722] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:14.731179 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7d7f7ee4384a2ba8bd0dac0ddad757 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=914b54d652844204bb800dbd8a5be98a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.731410 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7d7f7ee4384a2ba8bd0dac0ddad757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.731432 10888 tablet.cc:1620] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:14.731433 10887 tablet.cc:1620] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:14.737682 11104 tablet_bootstrap.cc:492] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:14.737671 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=ed7effdcb048462abc649fdf32202fa9] alter complete (version 2)
I20240503 07:59:14.738199 11102 tablet_bootstrap.cc:492] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:14.738504 9353 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7d7f7ee4384a2ba8bd0dac0ddad757 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=914b54d652844204bb800dbd8a5be98a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.738878 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7d7f7ee4384a2ba8bd0dac0ddad757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.740518 11101 tablet_bootstrap.cc:492] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:14.741102 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7d7f7ee4384a2ba8bd0dac0ddad757 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=914b54d652844204bb800dbd8a5be98a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.741439 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7d7f7ee4384a2ba8bd0dac0ddad757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.744055 6745 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 07:59:14.744192 6745 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 07:59:14.746785 6745 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 30e7227748a940e88984e92e959b4446
I20240503 07:59:14.747148 6745 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 1abaf2a559f94eb1a188b6263601e07a
I20240503 07:59:14.747485 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e7227748a940e88984e92e959b4446 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:14.747633 10998 tablet_bootstrap.cc:492] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:14.747691 10336 tablet_replica.cc:330] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:14.747730 10998 ts_tablet_manager.cc:1367] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240503 07:59:14.747800 6745 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 4 replicas of tablet 7f946926ddd84a2cb758a795638e3ef5
I20240503 07:59:14.747864 10336 raft_consensus.cc:2229] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.748086 10336 raft_consensus.cc:2258] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.748208 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e7227748a940e88984e92e959b4446 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:14.748239 10998 raft_consensus.cc:348] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.748361 10998 raft_consensus.cc:374] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.748391 10998 raft_consensus.cc:729] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.748536 10998 consensus_queue.cc:260] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.748664 10336 ts_tablet_manager.cc:1875] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.748834 10998 ts_tablet_manager.cc:1398] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.749033 10998 tablet_bootstrap.cc:492] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:14.749481 7130 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abaf2a559f94eb1a188b6263601e07a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:14.749673 10335 tablet_replica.cc:330] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:14.749815 10335 raft_consensus.cc:2229] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.749994 10998 tablet_bootstrap.cc:654] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.750064 10335 raft_consensus.cc:2258] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.750458 9957 tablet_replica.cc:330] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:14.750447 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e7227748a940e88984e92e959b4446 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:14.750555 9353 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f946926ddd84a2cb758a795638e3ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:14.750612 9957 raft_consensus.cc:2229] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.750703 10335 ts_tablet_manager.cc:1875] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.750747 10950 tablet_replica.cc:330] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:14.750751 7131 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f946926ddd84a2cb758a795638e3ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:14.750775 9957 raft_consensus.cc:2258] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.750887 10950 raft_consensus.cc:2229] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.750998 10950 raft_consensus.cc:2258] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.751295 9957 ts_tablet_manager.cc:1875] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.751641 10950 ts_tablet_manager.cc:1875] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.753185 10335 ts_tablet_manager.cc:1888] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.753270 10335 log.cc:1196] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1abaf2a559f94eb1a188b6263601e07a
I20240503 07:59:14.753544 10335 ts_tablet_manager.cc:1909] T 1abaf2a559f94eb1a188b6263601e07a P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:14.753782 9957 ts_tablet_manager.cc:1888] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.743778 11104 tablet_bootstrap.cc:654] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.753865 9957 log.cc:1196] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/30e7227748a940e88984e92e959b4446
I20240503 07:59:14.754233 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f946926ddd84a2cb758a795638e3ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:14.754302 10335 tablet_replica.cc:330] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:14.754503 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abaf2a559f94eb1a188b6263601e07a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:14.754573 11102 tablet_bootstrap.cc:654] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.754568 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abaf2a559f94eb1a188b6263601e07a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:14.754756 10947 tablet_replica.cc:330] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:14.754892 10947 raft_consensus.cc:2229] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.755043 10947 raft_consensus.cc:2258] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.755023 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 1abaf2a559f94eb1a188b6263601e07a (table rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]) successfully deleted
I20240503 07:59:14.755574 10947 ts_tablet_manager.cc:1875] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.755851 10335 raft_consensus.cc:2229] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.756004 10335 raft_consensus.cc:2258] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.756111 10950 ts_tablet_manager.cc:1888] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.756187 10950 log.cc:1196] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7f946926ddd84a2cb758a795638e3ef5
I20240503 07:59:14.756438 10950 ts_tablet_manager.cc:1909] T 7f946926ddd84a2cb758a795638e3ef5 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:14.756659 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f946926ddd84a2cb758a795638e3ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:14.756834 10335 ts_tablet_manager.cc:1875] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.757210 11101 tablet_bootstrap.cc:654] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.757215 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 7f946926ddd84a2cb758a795638e3ef5 (table rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]) successfully deleted
I20240503 07:59:14.758025 10947 ts_tablet_manager.cc:1888] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.758090 10947 log.cc:1196] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/1abaf2a559f94eb1a188b6263601e07a
I20240503 07:59:14.758426 11101 tablet_bootstrap.cc:492] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:14.758508 11101 ts_tablet_manager.cc:1367] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 07:59:14.759024 10335 ts_tablet_manager.cc:1888] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.759402 11101 raft_consensus.cc:348] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.759618 11101 raft_consensus.cc:374] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.759699 11101 raft_consensus.cc:729] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.760103 10336 ts_tablet_manager.cc:1888] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.760118 11101 consensus_queue.cc:260] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.760197 10336 log.cc:1196] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/30e7227748a940e88984e92e959b4446
I20240503 07:59:14.760440 10336 ts_tablet_manager.cc:1909] T 30e7227748a940e88984e92e959b4446 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:14.761337 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 30e7227748a940e88984e92e959b4446 (table rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]) successfully deleted
I20240503 07:59:14.761619 10336 tablet_replica.cc:330] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:14.761633 11101 ts_tablet_manager.cc:1398] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:14.761801 11101 tablet_bootstrap.cc:492] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:14.761826 10336 raft_consensus.cc:2229] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.761987 10336 raft_consensus.cc:2258] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.762514 10336 ts_tablet_manager.cc:1875] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.763664 11101 tablet_bootstrap.cc:654] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.763789 10335 log.cc:1196] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/30e7227748a940e88984e92e959b4446
I20240503 07:59:14.764137 10335 ts_tablet_manager.cc:1909] T 30e7227748a940e88984e92e959b4446 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:14.764801 11104 tablet_bootstrap.cc:492] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:14.764804 10335 tablet_replica.cc:330] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:14.764930 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 30e7227748a940e88984e92e959b4446 (table rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]) successfully deleted
I20240503 07:59:14.764963 10335 raft_consensus.cc:2229] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.764976 11104 ts_tablet_manager.cc:1367] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 07:59:14.765115 10335 raft_consensus.cc:2258] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.765223 10336 ts_tablet_manager.cc:1888] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.765302 10336 log.cc:1196] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7f946926ddd84a2cb758a795638e3ef5
I20240503 07:59:14.765476 9957 ts_tablet_manager.cc:1909] T 30e7227748a940e88984e92e959b4446 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:14.765602 10335 ts_tablet_manager.cc:1875] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.765651 11104 raft_consensus.cc:348] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.765779 11104 raft_consensus.cc:374] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.765841 11104 raft_consensus.cc:729] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.765987 11104 consensus_queue.cc:260] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.766216 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 30e7227748a940e88984e92e959b4446 (table rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]) successfully deleted
I20240503 07:59:14.766492 11104 ts_tablet_manager.cc:1398] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.767017 9957 tablet_replica.cc:330] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:14.767061 11101 tablet_bootstrap.cc:492] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:14.767179 9957 raft_consensus.cc:2229] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.767318 9957 raft_consensus.cc:2258] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.767834 9957 ts_tablet_manager.cc:1875] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.768479 11101 ts_tablet_manager.cc:1367] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:14.769318 11101 raft_consensus.cc:348] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.769510 11101 raft_consensus.cc:374] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.769580 11101 raft_consensus.cc:729] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.769785 11101 consensus_queue.cc:260] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.758320 10947 ts_tablet_manager.cc:1909] T 1abaf2a559f94eb1a188b6263601e07a P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:14.770689 11101 ts_tablet_manager.cc:1398] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:14.770924 11101 tablet_bootstrap.cc:492] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:14.771288 9957 ts_tablet_manager.cc:1888] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.771420 9957 log.cc:1196] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1abaf2a559f94eb1a188b6263601e07a
I20240503 07:59:14.771754 9957 ts_tablet_manager.cc:1909] T 1abaf2a559f94eb1a188b6263601e07a P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:14.771719 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 1abaf2a559f94eb1a188b6263601e07a (table rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]) successfully deleted
I20240503 07:59:14.772854 11101 tablet_bootstrap.cc:654] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.773257 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 1abaf2a559f94eb1a188b6263601e07a (table rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]) successfully deleted
I20240503 07:59:14.773677 10336 ts_tablet_manager.cc:1909] T 7f946926ddd84a2cb758a795638e3ef5 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:14.774473 9957 tablet_replica.cc:330] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:14.774601 11101 tablet_bootstrap.cc:492] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:14.774703 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 7f946926ddd84a2cb758a795638e3ef5 (table rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]) successfully deleted
I20240503 07:59:14.774869 11101 ts_tablet_manager.cc:1367] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:14.775436 11101 raft_consensus.cc:348] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:14.775941 9957 raft_consensus.cc:2229] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.776249 9957 raft_consensus.cc:2258] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.776304 11101 raft_consensus.cc:374] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.776391 11101 raft_consensus.cc:729] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.776599 11101 consensus_queue.cc:260] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:14.776933 9957 ts_tablet_manager.cc:1875] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.777650 11101 ts_tablet_manager.cc:1398] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:14.779389 9957 ts_tablet_manager.cc:1888] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.780251 10998 tablet_bootstrap.cc:492] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:14.780365 10998 ts_tablet_manager.cc:1367] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240503 07:59:14.780946 10998 raft_consensus.cc:348] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:14.781023 10335 ts_tablet_manager.cc:1888] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.781072 10998 raft_consensus.cc:374] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.781101 10335 log.cc:1196] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7f946926ddd84a2cb758a795638e3ef5
I20240503 07:59:14.781111 10998 raft_consensus.cc:729] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.781253 10998 consensus_queue.cc:260] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:14.781383 10335 ts_tablet_manager.cc:1909] T 7f946926ddd84a2cb758a795638e3ef5 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:14.781549 10998 ts_tablet_manager.cc:1398] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:14.781927 9957 log.cc:1196] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7f946926ddd84a2cb758a795638e3ef5
I20240503 07:59:14.782313 9957 ts_tablet_manager.cc:1909] T 7f946926ddd84a2cb758a795638e3ef5 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:14.782629 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 7f946926ddd84a2cb758a795638e3ef5 (table rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]) successfully deleted
I20240503 07:59:14.783779 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 7f946926ddd84a2cb758a795638e3ef5 (table rebalance_test_table_14 [id=86ee397c72104cc198e0a5e6d5cd38fe]) successfully deleted
I20240503 07:59:14.785483 11102 tablet_bootstrap.cc:492] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:14.785606 11102 ts_tablet_manager.cc:1367] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.000s
I20240503 07:59:14.786126 11102 raft_consensus.cc:348] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.786250 11102 raft_consensus.cc:374] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.786283 11102 raft_consensus.cc:729] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.786434 11102 consensus_queue.cc:260] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.786789 11102 ts_tablet_manager.cc:1398] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.786995 11102 tablet_bootstrap.cc:492] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:14.787945 11102 tablet_bootstrap.cc:654] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.792672 11102 tablet_bootstrap.cc:492] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:14.792799 11102 ts_tablet_manager.cc:1367] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:14.793326 11102 raft_consensus.cc:348] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:14.793443 11102 raft_consensus.cc:374] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.793469 11102 raft_consensus.cc:729] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.793583 11102 consensus_queue.cc:260] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:14.793859 11102 ts_tablet_manager.cc:1398] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.794035 11102 tablet_bootstrap.cc:492] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:14.795027 11102 tablet_bootstrap.cc:654] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.796193 11102 tablet_bootstrap.cc:492] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:14.796340 11102 ts_tablet_manager.cc:1367] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:14.796902 11102 raft_consensus.cc:348] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.797050 11102 raft_consensus.cc:374] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.797525 11102 raft_consensus.cc:729] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.798048 11102 consensus_queue.cc:260] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.798471 11102 ts_tablet_manager.cc:1398] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:14.799110 7131 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:14.810881 7260 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:14.811441 6999 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:14.817610 9353 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:14.819481 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:14.831228 6867 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:14.834966 9746 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:14.842924 6745 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:14.843092 11035 raft_consensus.cc:482] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:14.843240 11035 raft_consensus.cc:504] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.843830 11035 leader_election.cc:288] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:14.844218 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517c0e477a8c40fbb84c4cd3bb6ecabd" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:14.844286 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517c0e477a8c40fbb84c4cd3bb6ecabd" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:14.844383 9374 raft_consensus.cc:2454] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:14.844410 7278 raft_consensus.cc:2454] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:14.844693 6825 leader_election.cc:302] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:14.844970 11035 raft_consensus.cc:2781] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:14.845027 11035 raft_consensus.cc:482] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:14.845059 11035 raft_consensus.cc:3037] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.846274 11035 raft_consensus.cc:504] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.846706 11035 leader_election.cc:288] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:14.847020 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517c0e477a8c40fbb84c4cd3bb6ecabd" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:14.847167 7278 raft_consensus.cc:3037] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.847354 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517c0e477a8c40fbb84c4cd3bb6ecabd" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:14.847477 9374 raft_consensus.cc:3037] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.848408 7278 raft_consensus.cc:2454] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:14.848564 9374 raft_consensus.cc:2454] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:14.849293 6823 leader_election.cc:302] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:14.849562 11035 raft_consensus.cc:2781] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:14.849642 11035 raft_consensus.cc:686] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:14.851241 11035 consensus_queue.cc:237] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.851588 9485 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:14.852192 11051 tablet.cc:1722] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:14.853231 10887 tablet.cc:1722] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:14.853721 6745 catalog_manager.cc:5484] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:14.859838 10888 tablet.cc:1722] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:14.861956 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=ed7effdcb048462abc649fdf32202fa9] alter complete (version 3)
I20240503 07:59:14.916210 7278 consensus_queue.cc:237] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:14.917611 9504 raft_consensus.cc:1261] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.917666 6888 raft_consensus.cc:1261] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.918061 11141 consensus_queue.cc:1035] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.918272 10486 consensus_queue.cc:1035] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.920428 10486 raft_consensus.cc:2932] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:14.920991 9504 raft_consensus.cc:2932] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:14.922530 6888 raft_consensus.cc:2932] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:14.923663 6745 catalog_manager.cc:5484] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:14.927577 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc1fb490c6b448ab86bc707ee8faddc with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:14.927932 10336 tablet_replica.cc:330] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:14.928108 10336 raft_consensus.cc:2229] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.928267 10336 raft_consensus.cc:2258] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.928822 10336 ts_tablet_manager.cc:1875] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.930907 10336 ts_tablet_manager.cc:1888] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:14.930979 10336 log.cc:1196] T ebc1fb490c6b448ab86bc707ee8faddc P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ebc1fb490c6b448ab86bc707ee8faddc
I20240503 07:59:14.931505 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet ebc1fb490c6b448ab86bc707ee8faddc (table rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]) successfully deleted
I20240503 07:59:14.949390 10931 raft_consensus.cc:482] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:14.949544 10931 raft_consensus.cc:504] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.950060 10931 leader_election.cc:288] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:14.950488 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f59b9fe4a54e7d8beabce7589800b1" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:14.950492 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f59b9fe4a54e7d8beabce7589800b1" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:14.950641 9767 raft_consensus.cc:2454] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:14.950641 6888 raft_consensus.cc:2454] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:14.950948 9310 leader_election.cc:302] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:14.951126 10931 raft_consensus.cc:2781] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:14.951193 10931 raft_consensus.cc:482] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:14.951231 10931 raft_consensus.cc:3037] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.952425 10931 raft_consensus.cc:504] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.952850 10931 leader_election.cc:288] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:14.953246 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f59b9fe4a54e7d8beabce7589800b1" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:14.953253 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f59b9fe4a54e7d8beabce7589800b1" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:14.953380 9767 raft_consensus.cc:3037] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.953380 6888 raft_consensus.cc:3037] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.954460 9767 raft_consensus.cc:2454] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:14.954460 6888 raft_consensus.cc:2454] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:14.954798 9310 leader_election.cc:302] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:14.954998 10931 raft_consensus.cc:2781] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:14.955088 10931 raft_consensus.cc:686] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:14.955253 10931 consensus_queue.cc:237] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:14.957321 6741 catalog_manager.cc:5484] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:14.967339 6739 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:14.967501 6739 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:14.970630 6739 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 70abff3c68be431d8141ba7bfc5866d2
I20240503 07:59:14.971454 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 70abff3c68be431d8141ba7bfc5866d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:14.971668 10336 tablet_replica.cc:330] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:14.971733 9746 tablet_service.cc:1514] Processing DeleteTablet for tablet 70abff3c68be431d8141ba7bfc5866d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:14.971863 10336 raft_consensus.cc:2229] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.972044 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet 70abff3c68be431d8141ba7bfc5866d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:14.972107 10336 raft_consensus.cc:2258] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.972483 11168 tablet_replica.cc:330] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:14.972597 11168 raft_consensus.cc:2229] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.972712 11168 raft_consensus.cc:2258] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.973330 11168 ts_tablet_manager.cc:1875] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.976279 11168 ts_tablet_manager.cc:1888] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.976418 11168 log.cc:1196] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/70abff3c68be431d8141ba7bfc5866d2
I20240503 07:59:14.976670 11168 ts_tablet_manager.cc:1909] T 70abff3c68be431d8141ba7bfc5866d2 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:14.977680 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 70abff3c68be431d8141ba7bfc5866d2 (table rebalancer_dynamic_table_new_name [id=ed7effdcb048462abc649fdf32202fa9]) successfully deleted
I20240503 07:59:14.978544 11167 tablet_replica.cc:330] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:14.978549 10336 ts_tablet_manager.cc:1875] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.978834 11167 raft_consensus.cc:2229] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.979027 11167 raft_consensus.cc:2258] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.979734 11167 ts_tablet_manager.cc:1875] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.981724 10336 ts_tablet_manager.cc:1888] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.981814 10336 log.cc:1196] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/70abff3c68be431d8141ba7bfc5866d2
I20240503 07:59:14.982153 10336 ts_tablet_manager.cc:1909] T 70abff3c68be431d8141ba7bfc5866d2 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:14.982443 11167 ts_tablet_manager.cc:1888] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.982578 11167 log.cc:1196] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/70abff3c68be431d8141ba7bfc5866d2
I20240503 07:59:14.982928 11167 ts_tablet_manager.cc:1909] T 70abff3c68be431d8141ba7bfc5866d2 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:14.983673 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 70abff3c68be431d8141ba7bfc5866d2 (table rebalancer_dynamic_table_new_name [id=ed7effdcb048462abc649fdf32202fa9]) successfully deleted
I20240503 07:59:14.983839 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 70abff3c68be431d8141ba7bfc5866d2 (table rebalancer_dynamic_table_new_name [id=ed7effdcb048462abc649fdf32202fa9]) successfully deleted
I20240503 07:59:15.017288 10166 raft_consensus.cc:482] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:15.017439 10166 raft_consensus.cc:504] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:15.018041 10166 leader_election.cc:288] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:15.018419 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7d7f7ee4384a2ba8bd0dac0ddad757" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:15.018579 9374 raft_consensus.cc:2454] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:15.018594 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7d7f7ee4384a2ba8bd0dac0ddad757" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:15.018734 6886 raft_consensus.cc:2454] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:15.019047 9701 leader_election.cc:302] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:15.019804 10166 raft_consensus.cc:2781] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:15.019883 10166 raft_consensus.cc:482] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:15.019914 10166 raft_consensus.cc:3037] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.020951 10166 raft_consensus.cc:504] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:15.021360 10166 leader_election.cc:288] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:15.021806 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7d7f7ee4384a2ba8bd0dac0ddad757" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:15.021963 6886 raft_consensus.cc:3037] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.021950 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7d7f7ee4384a2ba8bd0dac0ddad757" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:15.022064 9374 raft_consensus.cc:3037] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.023104 6886 raft_consensus.cc:2454] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:15.023104 9374 raft_consensus.cc:2454] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:15.023505 9700 leader_election.cc:302] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:15.023694 10166 raft_consensus.cc:2781] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:15.023866 10166 raft_consensus.cc:686] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:15.024025 10166 consensus_queue.cc:237] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:15.026624 6738 catalog_manager.cc:5484] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:15.045027 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:15.046756 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:15.046844 6999 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:15.049346 7260 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:15.059768 9485 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:15.062870 6867 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:15.063190 9746 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:15.072278 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:15.086946 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0b68d6e42e4a18936d48ff583dacfb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5d9b659f45574affb021962f285f3f42]), partition=
I20240503 07:59:15.087352 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0b68d6e42e4a18936d48ff583dacfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.087798 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0b68d6e42e4a18936d48ff583dacfb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5d9b659f45574affb021962f285f3f42]), partition=
I20240503 07:59:15.088068 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0b68d6e42e4a18936d48ff583dacfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.088546 9614 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0b68d6e42e4a18936d48ff583dacfb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5d9b659f45574affb021962f285f3f42]), partition=
I20240503 07:59:15.088819 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0b68d6e42e4a18936d48ff583dacfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.090369 11200 tablet_bootstrap.cc:492] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:15.091758 11200 tablet_bootstrap.cc:654] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.093135 11202 tablet_bootstrap.cc:492] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:15.093225 11200 tablet_bootstrap.cc:492] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:15.093350 11200 ts_tablet_manager.cc:1367] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:15.094959 11202 tablet_bootstrap.cc:654] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.094941 11200 raft_consensus.cc:348] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.096509 11200 raft_consensus.cc:374] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.096624 11204 tablet_bootstrap.cc:492] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:15.096665 11200 raft_consensus.cc:729] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.096731 11202 tablet_bootstrap.cc:492] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:15.096902 11202 ts_tablet_manager.cc:1367] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:15.096859 11200 consensus_queue.cc:260] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.097185 11200 ts_tablet_manager.cc:1398] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:15.098079 11202 raft_consensus.cc:348] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.098284 11202 raft_consensus.cc:374] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.098398 11202 raft_consensus.cc:729] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.098590 11202 consensus_queue.cc:260] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.099061 11202 ts_tablet_manager.cc:1398] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:15.102526 7131 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:15.104386 11204 tablet_bootstrap.cc:654] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.105579 11204 tablet_bootstrap.cc:492] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:15.105700 11204 ts_tablet_manager.cc:1367] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:15.106140 11204 raft_consensus.cc:348] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.106266 11204 raft_consensus.cc:374] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.106297 11204 raft_consensus.cc:729] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.106731 11204 consensus_queue.cc:260] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.107764 11204 ts_tablet_manager.cc:1398] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:15.157835 7019 consensus_queue.cc:237] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.159372 7149 raft_consensus.cc:1261] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.159387 9504 raft_consensus.cc:1261] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.159857 11100 consensus_queue.cc:1035] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.160056 9151 consensus_queue.cc:1035] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.161929 10310 raft_consensus.cc:2932] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:15.162459 7149 raft_consensus.cc:2932] T 82fc9f4263244c6fa2fb39543230eb87 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:15.162551 9504 raft_consensus.cc:2932] T 82fc9f4263244c6fa2fb39543230eb87 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:15.165228 6738 catalog_manager.cc:5484] T 82fc9f4263244c6fa2fb39543230eb87 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:15.169095 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 82fc9f4263244c6fa2fb39543230eb87 with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:15.169428 10947 tablet_replica.cc:330] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:15.169601 10947 raft_consensus.cc:2229] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.169759 10947 raft_consensus.cc:2258] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.170378 10947 ts_tablet_manager.cc:1875] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:15.172641 10947 ts_tablet_manager.cc:1888] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:15.172734 10947 log.cc:1196] T 82fc9f4263244c6fa2fb39543230eb87 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/82fc9f4263244c6fa2fb39543230eb87
I20240503 07:59:15.173270 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 82fc9f4263244c6fa2fb39543230eb87 (table rebalance_test_table_19 [id=071fb84151514ba094acc7ff3c1b24e7]) successfully deleted
I20240503 07:59:15.236729 10651 consensus_queue.cc:1035] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.242374 10651 consensus_queue.cc:1035] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.249975 10716 raft_consensus.cc:482] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:15.250128 10716 raft_consensus.cc:504] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.250639 10716 leader_election.cc:288] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:15.251425 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0b68d6e42e4a18936d48ff583dacfb" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:15.251592 9635 raft_consensus.cc:2454] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:15.252655 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0b68d6e42e4a18936d48ff583dacfb" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:15.252815 9504 raft_consensus.cc:2454] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:15.253127 7084 leader_election.cc:302] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2; no voters:
I20240503 07:59:15.254532 11058 raft_consensus.cc:2781] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:15.254606 11058 raft_consensus.cc:482] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:15.254645 11058 raft_consensus.cc:3037] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.256588 11058 raft_consensus.cc:504] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.256983 11058 leader_election.cc:288] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:15.257472 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0b68d6e42e4a18936d48ff583dacfb" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:15.257611 9635 raft_consensus.cc:3037] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.258558 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0b68d6e42e4a18936d48ff583dacfb" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:15.258702 9504 raft_consensus.cc:3037] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.259837 9504 raft_consensus.cc:2454] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:15.260177 7087 leader_election.cc:302] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:15.260360 11188 raft_consensus.cc:2781] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:15.260418 11188 raft_consensus.cc:686] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:15.260582 11188 consensus_queue.cc:237] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.262766 6745 catalog_manager.cc:5484] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:15.264015 9635 raft_consensus.cc:2454] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:15.265026 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 07:59:15.265157 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 07:59:15.267645 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet ebc1fb490c6b448ab86bc707ee8faddc
I20240503 07:59:15.268253 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc1fb490c6b448ab86bc707ee8faddc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:15.268277 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc1fb490c6b448ab86bc707ee8faddc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:15.268453 10947 tablet_replica.cc:330] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:15.268465 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc1fb490c6b448ab86bc707ee8faddc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:15.268661 11168 tablet_replica.cc:330] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:15.268658 10947 raft_consensus.cc:2229] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:15.268734 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 1564dc8b294e4c7da812cd4fa6d510da
I20240503 07:59:15.268807 11168 raft_consensus.cc:2229] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.268885 10947 raft_consensus.cc:2258] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.268925 11168 raft_consensus.cc:2258] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.269078 9957 tablet_replica.cc:330] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:15.269232 9957 raft_consensus.cc:2229] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.269296 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 4 replicas of tablet a00e3d35dbe5490f86bccd96a97f92f7
I20240503 07:59:15.269424 9957 raft_consensus.cc:2258] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.269779 7131 tablet_service.cc:1514] Processing DeleteTablet for tablet 1564dc8b294e4c7da812cd4fa6d510da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:15.269806 10947 ts_tablet_manager.cc:1875] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.269932 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet a00e3d35dbe5490f86bccd96a97f92f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:15.270262 11168 ts_tablet_manager.cc:1875] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.271437 9614 tablet_service.cc:1514] Processing DeleteTablet for tablet a00e3d35dbe5490f86bccd96a97f92f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:15.271764 11269 tablet_replica.cc:330] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:15.271903 11269 raft_consensus.cc:2229] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.272014 11269 raft_consensus.cc:2258] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.272388 10947 ts_tablet_manager.cc:1888] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 07:59:15.272388 11168 ts_tablet_manager.cc:1888] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 07:59:15.272472 10947 log.cc:1196] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ebc1fb490c6b448ab86bc707ee8faddc
I20240503 07:59:15.272472 11168 log.cc:1196] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/ebc1fb490c6b448ab86bc707ee8faddc
I20240503 07:59:15.272524 11269 ts_tablet_manager.cc:1875] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.272732 10947 ts_tablet_manager.cc:1909] T ebc1fb490c6b448ab86bc707ee8faddc P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:15.272735 11168 ts_tablet_manager.cc:1909] T ebc1fb490c6b448ab86bc707ee8faddc P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:15.272960 10335 tablet_replica.cc:330] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:15.273113 10335 raft_consensus.cc:2229] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:15.273207 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet a00e3d35dbe5490f86bccd96a97f92f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:15.273350 10335 raft_consensus.cc:2258] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.273367 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet 1564dc8b294e4c7da812cd4fa6d510da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:15.273929 10335 ts_tablet_manager.cc:1875] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.274159 9957 ts_tablet_manager.cc:1875] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.274550 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 1564dc8b294e4c7da812cd4fa6d510da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:15.274747 10336 tablet_replica.cc:330] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:15.274890 10336 raft_consensus.cc:2229] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.274909 11269 ts_tablet_manager.cc:1888] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.274967 11269 log.cc:1196] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a00e3d35dbe5490f86bccd96a97f92f7
I20240503 07:59:15.275036 10336 raft_consensus.cc:2258] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.275221 11269 ts_tablet_manager.cc:1909] T a00e3d35dbe5490f86bccd96a97f92f7 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:15.275514 10336 ts_tablet_manager.cc:1875] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.276084 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet a00e3d35dbe5490f86bccd96a97f92f7 (table rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]) successfully deleted
I20240503 07:59:15.276294 10335 ts_tablet_manager.cc:1888] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:15.276365 10335 log.cc:1196] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1564dc8b294e4c7da812cd4fa6d510da
I20240503 07:59:15.276672 9957 ts_tablet_manager.cc:1888] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 07:59:15.276744 9957 log.cc:1196] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ebc1fb490c6b448ab86bc707ee8faddc
I20240503 07:59:15.276995 9957 ts_tablet_manager.cc:1909] T ebc1fb490c6b448ab86bc707ee8faddc P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:15.277546 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet a00e3d35dbe5490f86bccd96a97f92f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:15.277714 10336 ts_tablet_manager.cc:1888] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:15.277779 10336 log.cc:1196] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1564dc8b294e4c7da812cd4fa6d510da
I20240503 07:59:15.277791 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet ebc1fb490c6b448ab86bc707ee8faddc (table rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]) successfully deleted
I20240503 07:59:15.278524 10336 ts_tablet_manager.cc:1909] T 1564dc8b294e4c7da812cd4fa6d510da P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:15.278550 9957 tablet_replica.cc:330] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:15.278687 9957 raft_consensus.cc:2229] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.278834 9957 raft_consensus.cc:2258] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.279312 9957 ts_tablet_manager.cc:1875] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.280252 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet ebc1fb490c6b448ab86bc707ee8faddc (table rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]) successfully deleted
I20240503 07:59:15.281373 9957 ts_tablet_manager.cc:1888] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.281448 9957 log.cc:1196] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a00e3d35dbe5490f86bccd96a97f92f7
I20240503 07:59:15.281678 9957 ts_tablet_manager.cc:1909] T a00e3d35dbe5490f86bccd96a97f92f7 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:15.282563 10947 tablet_replica.cc:330] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:15.282765 10947 raft_consensus.cc:2229] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:15.283020 10947 raft_consensus.cc:2258] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.283578 10947 ts_tablet_manager.cc:1875] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.286214 10947 ts_tablet_manager.cc:1888] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.286306 10947 log.cc:1196] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a00e3d35dbe5490f86bccd96a97f92f7
I20240503 07:59:15.286581 10947 ts_tablet_manager.cc:1909] T a00e3d35dbe5490f86bccd96a97f92f7 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:15.287405 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet a00e3d35dbe5490f86bccd96a97f92f7 (table rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]) successfully deleted
I20240503 07:59:15.287864 9957 tablet_replica.cc:330] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:15.288100 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 1564dc8b294e4c7da812cd4fa6d510da (table rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]) successfully deleted
I20240503 07:59:15.288355 9957 raft_consensus.cc:2229] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.288748 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet ebc1fb490c6b448ab86bc707ee8faddc (table rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]) successfully deleted
I20240503 07:59:15.289055 9957 raft_consensus.cc:2258] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.289178 10335 ts_tablet_manager.cc:1909] T 1564dc8b294e4c7da812cd4fa6d510da P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:15.290134 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet a00e3d35dbe5490f86bccd96a97f92f7 (table rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]) successfully deleted
I20240503 07:59:15.290278 9957 ts_tablet_manager.cc:1875] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.291270 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 1564dc8b294e4c7da812cd4fa6d510da (table rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]) successfully deleted
I20240503 07:59:15.291162 10335 tablet_replica.cc:330] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:15.292892 10335 raft_consensus.cc:2229] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.293098 10335 raft_consensus.cc:2258] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.293680 10335 ts_tablet_manager.cc:1875] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.294430 9957 ts_tablet_manager.cc:1888] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:15.294591 9957 log.cc:1196] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1564dc8b294e4c7da812cd4fa6d510da
I20240503 07:59:15.294961 9957 ts_tablet_manager.cc:1909] T 1564dc8b294e4c7da812cd4fa6d510da P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:15.295809 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 1564dc8b294e4c7da812cd4fa6d510da (table rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]) successfully deleted
I20240503 07:59:15.296394 10335 ts_tablet_manager.cc:1888] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.296478 10335 log.cc:1196] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a00e3d35dbe5490f86bccd96a97f92f7
I20240503 07:59:15.296743 10335 ts_tablet_manager.cc:1909] T a00e3d35dbe5490f86bccd96a97f92f7 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:15.297755 7129 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:15.300320 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:15.301183 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet a00e3d35dbe5490f86bccd96a97f92f7 (table rebalance_test_table_16 [id=2e16e624bea24c498601622e1f767652]) successfully deleted
I20240503 07:59:15.303139 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:15.313962 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:15.321501 9746 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:15.322000 6999 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:15.336267 9485 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:15.336464 6867 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:15.389943 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:15.396559 9504 raft_consensus.cc:1261] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:15.396584 9635 raft_consensus.cc:1261] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:15.397100 10716 consensus_queue.cc:1035] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.400971 6740 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 1c0b68d6e42e4a18936d48ff583dacfb (table rebalancer_dynamic_table [id=5d9b659f45574affb021962f285f3f42]). Expected version 1 got 0
I20240503 07:59:15.402310 11295 tablet.cc:1722] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:15.403040 11295 tablet.cc:1620] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:15.402930 11296 tablet.cc:1722] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:15.403651 11058 consensus_queue.cc:1035] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.404055 11296 tablet.cc:1620] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:15.404278 6740 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 1c0b68d6e42e4a18936d48ff583dacfb (table rebalancer_dynamic_table [id=5d9b659f45574affb021962f285f3f42]). Expected version 1 got 0
I20240503 07:59:15.407495 6745 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 1c0b68d6e42e4a18936d48ff583dacfb (table rebalancer_dynamic_table [id=5d9b659f45574affb021962f285f3f42]). Expected version 1 got 0
I20240503 07:59:15.409781 11300 tablet.cc:1722] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:15.409914 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=5d9b659f45574affb021962f285f3f42] alter complete (version 1)
I20240503 07:59:15.410235 11300 tablet.cc:1620] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:15.478128 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:15.478611 7129 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:15.483144 6867 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:15.497320 6999 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:15.499809 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:15.503151 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:15.509042 9485 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:15.511061 9746 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:15.511445 10931 consensus_queue.cc:1035] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.514873 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:15.527422 10694 consensus_queue.cc:1035] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.530402 11295 tablet.cc:1722] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:15.530803 11295 tablet.cc:1620] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:15.531904 11296 tablet.cc:1722] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:15.532263 11296 tablet.cc:1620] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:15.541821 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=5d9b659f45574affb021962f285f3f42] alter complete (version 2)
I20240503 07:59:15.544229 10166 consensus_queue.cc:1035] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.545780 11300 tablet.cc:1722] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:15.546161 11300 tablet.cc:1620] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:15.548970 10166 consensus_queue.cc:1035] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.572429 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:15.584324 6999 tablet_service.cc:1467] Processing CreateTablet for tablet da9b313c416a49239cb266f90445ec4a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=fa3c77fefb014f88b3cad9f4ea921534]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.584532 9353 tablet_service.cc:1467] Processing CreateTablet for tablet df569029c1514bc5ae4d77327cd03478 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=fa3c77fefb014f88b3cad9f4ea921534]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.584720 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9b313c416a49239cb266f90445ec4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.584770 9485 tablet_service.cc:1467] Processing CreateTablet for tablet da9b313c416a49239cb266f90445ec4a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=fa3c77fefb014f88b3cad9f4ea921534]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.584838 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df569029c1514bc5ae4d77327cd03478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.585037 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9b313c416a49239cb266f90445ec4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.587500 9484 tablet_service.cc:1467] Processing CreateTablet for tablet df569029c1514bc5ae4d77327cd03478 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=fa3c77fefb014f88b3cad9f4ea921534]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.587792 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df569029c1514bc5ae4d77327cd03478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.590497 11200 tablet_bootstrap.cc:492] T da9b313c416a49239cb266f90445ec4a P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:15.591564 11200 tablet_bootstrap.cc:654] T da9b313c416a49239cb266f90445ec4a P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.593279 9614 tablet_service.cc:1467] Processing CreateTablet for tablet cf8e0a0da8b941e28e797de8f8b7a3f4 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=fa3c77fefb014f88b3cad9f4ea921534]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.593595 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf8e0a0da8b941e28e797de8f8b7a3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.594688 6867 tablet_service.cc:1467] Processing CreateTablet for tablet cf8e0a0da8b941e28e797de8f8b7a3f4 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=fa3c77fefb014f88b3cad9f4ea921534]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.594708 9615 tablet_service.cc:1467] Processing CreateTablet for tablet df569029c1514bc5ae4d77327cd03478 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=fa3c77fefb014f88b3cad9f4ea921534]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.594971 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df569029c1514bc5ae4d77327cd03478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.594991 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf8e0a0da8b941e28e797de8f8b7a3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.597334 11204 tablet_bootstrap.cc:492] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:15.597973 11352 tablet_bootstrap.cc:492] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:15.598368 11204 tablet_bootstrap.cc:654] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.599282 11352 tablet_bootstrap.cc:654] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.600364 11352 tablet_bootstrap.cc:492] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:15.600472 11352 ts_tablet_manager.cc:1367] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:15.600979 11352 raft_consensus.cc:348] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:15.601099 11352 raft_consensus.cc:374] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.601141 11352 raft_consensus.cc:729] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.601275 11352 consensus_queue.cc:260] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:15.601578 11352 ts_tablet_manager.cc:1398] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.602180 7131 tablet_service.cc:1467] Processing CreateTablet for tablet da9b313c416a49239cb266f90445ec4a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=fa3c77fefb014f88b3cad9f4ea921534]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.602535 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9b313c416a49239cb266f90445ec4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.602766 7129 tablet_service.cc:1467] Processing CreateTablet for tablet cf8e0a0da8b941e28e797de8f8b7a3f4 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=fa3c77fefb014f88b3cad9f4ea921534]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.603032 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf8e0a0da8b941e28e797de8f8b7a3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.605978 11355 tablet_bootstrap.cc:492] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:15.607681 11355 tablet_bootstrap.cc:654] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.608772 11350 tablet_bootstrap.cc:492] T da9b313c416a49239cb266f90445ec4a P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:15.609624 11200 tablet_bootstrap.cc:492] T da9b313c416a49239cb266f90445ec4a P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:15.609738 11200 ts_tablet_manager.cc:1367] T da9b313c416a49239cb266f90445ec4a P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 07:59:15.610275 11200 raft_consensus.cc:348] T da9b313c416a49239cb266f90445ec4a P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:15.611959 11200 raft_consensus.cc:374] T da9b313c416a49239cb266f90445ec4a P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.612006 11200 raft_consensus.cc:729] T da9b313c416a49239cb266f90445ec4a P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.612144 11200 consensus_queue.cc:260] T da9b313c416a49239cb266f90445ec4a P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:15.612460 11200 ts_tablet_manager.cc:1398] T da9b313c416a49239cb266f90445ec4a P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:15.612681 11200 tablet_bootstrap.cc:492] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:15.613772 11200 tablet_bootstrap.cc:654] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.616639 11355 tablet_bootstrap.cc:492] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:15.616751 11355 ts_tablet_manager.cc:1367] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:15.617624 11355 raft_consensus.cc:348] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:15.617776 11355 raft_consensus.cc:374] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.617810 11355 raft_consensus.cc:729] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.617959 11355 consensus_queue.cc:260] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:15.617990 11204 tablet_bootstrap.cc:492] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:15.618103 11204 ts_tablet_manager.cc:1367] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.002s
I20240503 07:59:15.618302 11355 ts_tablet_manager.cc:1398] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.618587 11355 tablet_bootstrap.cc:492] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:15.618842 11204 raft_consensus.cc:348] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.618983 11204 raft_consensus.cc:374] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.619015 11204 raft_consensus.cc:729] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.619158 11204 consensus_queue.cc:260] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.619480 11204 ts_tablet_manager.cc:1398] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:15.619676 11204 tablet_bootstrap.cc:492] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:15.620009 11355 tablet_bootstrap.cc:654] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.620754 11204 tablet_bootstrap.cc:654] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.623998 11204 tablet_bootstrap.cc:492] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:15.624156 11204 ts_tablet_manager.cc:1367] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 07:59:15.624912 11204 raft_consensus.cc:348] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:15.625056 11204 raft_consensus.cc:374] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.625104 11204 raft_consensus.cc:729] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.625272 11204 consensus_queue.cc:260] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:15.626004 11204 ts_tablet_manager.cc:1398] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:15.627597 11350 tablet_bootstrap.cc:654] T da9b313c416a49239cb266f90445ec4a P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.629004 11200 tablet_bootstrap.cc:492] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:15.629125 11200 ts_tablet_manager.cc:1367] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240503 07:59:15.629688 11200 raft_consensus.cc:348] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.629818 11200 raft_consensus.cc:374] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.629853 11200 raft_consensus.cc:729] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.629998 11200 consensus_queue.cc:260] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.630313 11200 ts_tablet_manager.cc:1398] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:15.631580 11350 tablet_bootstrap.cc:492] T da9b313c416a49239cb266f90445ec4a P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:15.631778 11350 ts_tablet_manager.cc:1367] T da9b313c416a49239cb266f90445ec4a P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240503 07:59:15.632426 11350 raft_consensus.cc:348] T da9b313c416a49239cb266f90445ec4a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:15.632895 11350 raft_consensus.cc:374] T da9b313c416a49239cb266f90445ec4a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.632987 11350 raft_consensus.cc:729] T da9b313c416a49239cb266f90445ec4a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.633215 11350 consensus_queue.cc:260] T da9b313c416a49239cb266f90445ec4a P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:15.633641 11350 ts_tablet_manager.cc:1398] T da9b313c416a49239cb266f90445ec4a P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:15.634192 11349 tablet_bootstrap.cc:492] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:15.635486 11349 tablet_bootstrap.cc:654] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.636900 11355 tablet_bootstrap.cc:492] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:15.637079 11355 ts_tablet_manager.cc:1367] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240503 07:59:15.638382 11355 raft_consensus.cc:348] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:15.638518 11355 raft_consensus.cc:374] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.638576 11355 raft_consensus.cc:729] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.638716 11355 consensus_queue.cc:260] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:15.639104 11355 ts_tablet_manager.cc:1398] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:15.643504 11349 tablet_bootstrap.cc:492] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:15.643726 11349 ts_tablet_manager.cc:1367] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 07:59:15.644299 11349 raft_consensus.cc:348] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.644443 11349 raft_consensus.cc:374] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.644515 11349 raft_consensus.cc:729] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.644678 11349 consensus_queue.cc:260] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.645624 11349 ts_tablet_manager.cc:1398] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:15.646778 11347 raft_consensus.cc:482] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:15.646890 11347 raft_consensus.cc:504] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:15.647420 11347 leader_election.cc:288] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:15.647872 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8e0a0da8b941e28e797de8f8b7a3f4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:15.648025 9635 raft_consensus.cc:2454] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:15.648231 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8e0a0da8b941e28e797de8f8b7a3f4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:15.648356 7149 raft_consensus.cc:2454] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:15.648480 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:15.648546 6822 leader_election.cc:302] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:15.648793 11347 raft_consensus.cc:2781] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:15.648844 11347 raft_consensus.cc:482] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:15.648877 11347 raft_consensus.cc:3037] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.650004 11347 raft_consensus.cc:504] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:15.650439 11347 leader_election.cc:288] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:15.651194 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8e0a0da8b941e28e797de8f8b7a3f4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:15.651324 7149 raft_consensus.cc:3037] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.651736 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8e0a0da8b941e28e797de8f8b7a3f4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:15.651866 9635 raft_consensus.cc:3037] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.652451 7149 raft_consensus.cc:2454] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:15.652880 9635 raft_consensus.cc:2454] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:15.653277 6822 leader_election.cc:302] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:15.653539 11347 raft_consensus.cc:2781] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:15.653628 11347 raft_consensus.cc:686] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:15.653772 11347 consensus_queue.cc:237] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:15.656921 11295 tablet.cc:1722] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:15.656877 6738 catalog_manager.cc:5484] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:15.657342 11300 tablet.cc:1722] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:15.658624 11296 tablet.cc:1722] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:15.663854 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=5d9b659f45574affb021962f285f3f42] alter complete (version 3)
I20240503 07:59:15.665390 7019 consensus_queue.cc:237] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:15.666790 6886 raft_consensus.cc:1261] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.666829 9635 raft_consensus.cc:1261] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.667210 10310 consensus_queue.cc:1035] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.667414 9151 consensus_queue.cc:1035] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.669268 9151 raft_consensus.cc:2932] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:15.669765 9635 raft_consensus.cc:2932] T 7b424a8b7bba44b098731f8ee18e8c9b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:15.670946 6886 raft_consensus.cc:2932] T 7b424a8b7bba44b098731f8ee18e8c9b P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:15.672351 6742 catalog_manager.cc:5484] T 7b424a8b7bba44b098731f8ee18e8c9b P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:15.672991 11299 raft_consensus.cc:482] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:15.673128 11299 raft_consensus.cc:504] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.673676 11299 leader_election.cc:288] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:15.674050 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df569029c1514bc5ae4d77327cd03478" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:15.674202 9635 raft_consensus.cc:2454] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:15.674482 9438 leader_election.cc:302] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:15.675629 11299 raft_consensus.cc:2781] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:15.675693 11299 raft_consensus.cc:482] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:15.675730 11299 raft_consensus.cc:3037] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.676952 11299 raft_consensus.cc:504] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.677316 11299 leader_election.cc:288] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:15.677946 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b424a8b7bba44b098731f8ee18e8c9b with delete_type TABLET_DATA_TOMBSTONED (TS 00a3f8e159c64327832d79fb32cdcca8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:15.678184 10335 tablet_replica.cc:330] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:15.678378 10335 raft_consensus.cc:2229] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.678544 10335 raft_consensus.cc:2258] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.678781 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df569029c1514bc5ae4d77327cd03478" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:15.678923 9635 raft_consensus.cc:3037] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.679149 10335 ts_tablet_manager.cc:1875] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:15.680371 9635 raft_consensus.cc:2454] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:15.680675 9438 leader_election.cc:302] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:15.681053 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df569029c1514bc5ae4d77327cd03478" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:15.681200 9374 raft_consensus.cc:2454] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:15.681708 10335 ts_tablet_manager.cc:1888] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:15.681763 10335 log.cc:1196] T 7b424a8b7bba44b098731f8ee18e8c9b P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7b424a8b7bba44b098731f8ee18e8c9b
I20240503 07:59:15.682519 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df569029c1514bc5ae4d77327cd03478" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:15.682623 9357 raft_consensus.cc:3037] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.683531 9357 raft_consensus.cc:2454] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:15.684265 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 7b424a8b7bba44b098731f8ee18e8c9b (table rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]) successfully deleted
I20240503 07:59:15.685215 11299 raft_consensus.cc:2781] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:15.685344 11299 raft_consensus.cc:686] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:15.685503 11299 consensus_queue.cc:237] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.687525 6746 catalog_manager.cc:5484] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:15.765548 6741 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:15.765702 6741 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:15.767807 6741 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 1c0b68d6e42e4a18936d48ff583dacfb
I20240503 07:59:15.768355 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c0b68d6e42e4a18936d48ff583dacfb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:15.768407 9614 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c0b68d6e42e4a18936d48ff583dacfb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:15.768548 10335 tablet_replica.cc:330] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:15.768685 10335 raft_consensus.cc:2229] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:15.768919 10335 raft_consensus.cc:2258] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.769146 11269 tablet_replica.cc:330] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:15.769282 11269 raft_consensus.cc:2229] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.769395 11269 raft_consensus.cc:2258] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.769454 10716 raft_consensus.cc:482] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:15.769541 10716 raft_consensus.cc:504] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:15.770059 10716 leader_election.cc:288] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:15.770113 11269 ts_tablet_manager.cc:1875] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.770138 10335 ts_tablet_manager.cc:1875] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.770506 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9b313c416a49239cb266f90445ec4a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:15.770641 9504 raft_consensus.cc:2454] T da9b313c416a49239cb266f90445ec4a P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:15.770951 7087 leader_election.cc:302] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:15.771194 10716 raft_consensus.cc:2781] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:15.771260 10716 raft_consensus.cc:482] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:15.771291 10716 raft_consensus.cc:3037] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.772218 10716 raft_consensus.cc:504] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:15.772547 10716 leader_election.cc:288] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:15.772589 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9b313c416a49239cb266f90445ec4a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:15.772743 7019 raft_consensus.cc:2454] T da9b313c416a49239cb266f90445ec4a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:15.773329 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9b313c416a49239cb266f90445ec4a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:15.773447 9504 raft_consensus.cc:3037] T da9b313c416a49239cb266f90445ec4a P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.773546 9484 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c0b68d6e42e4a18936d48ff583dacfb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:15.773578 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9b313c416a49239cb266f90445ec4a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:15.773675 7019 raft_consensus.cc:3037] T da9b313c416a49239cb266f90445ec4a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.773737 11168 tablet_replica.cc:330] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:15.773849 10335 ts_tablet_manager.cc:1888] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.773867 11168 raft_consensus.cc:2229] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.773906 10335 log.cc:1196] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1c0b68d6e42e4a18936d48ff583dacfb
I20240503 07:59:15.773983 11168 raft_consensus.cc:2258] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.774180 10335 ts_tablet_manager.cc:1909] T 1c0b68d6e42e4a18936d48ff583dacfb P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:15.774458 9504 raft_consensus.cc:2454] T da9b313c416a49239cb266f90445ec4a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:15.774677 7019 raft_consensus.cc:2454] T da9b313c416a49239cb266f90445ec4a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:15.774684 11168 ts_tablet_manager.cc:1875] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.774771 7087 leader_election.cc:302] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:15.774986 11058 raft_consensus.cc:2781] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:15.775112 11058 raft_consensus.cc:686] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:15.775233 11058 consensus_queue.cc:237] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:15.775326 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 1c0b68d6e42e4a18936d48ff583dacfb (table rebalancer_dynamic_table_new_name [id=5d9b659f45574affb021962f285f3f42]) successfully deleted
I20240503 07:59:15.775599 11269 ts_tablet_manager.cc:1888] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.775676 11269 log.cc:1196] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/1c0b68d6e42e4a18936d48ff583dacfb
I20240503 07:59:15.775960 11269 ts_tablet_manager.cc:1909] T 1c0b68d6e42e4a18936d48ff583dacfb P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:15.776696 11168 ts_tablet_manager.cc:1888] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.776765 11168 log.cc:1196] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/1c0b68d6e42e4a18936d48ff583dacfb
I20240503 07:59:15.776988 11168 ts_tablet_manager.cc:1909] T 1c0b68d6e42e4a18936d48ff583dacfb P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:15.777282 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 1c0b68d6e42e4a18936d48ff583dacfb (table rebalancer_dynamic_table_new_name [id=5d9b659f45574affb021962f285f3f42]) successfully deleted
I20240503 07:59:15.777555 6741 catalog_manager.cc:5484] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:15.777678 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 1c0b68d6e42e4a18936d48ff583dacfb (table rebalancer_dynamic_table_new_name [id=5d9b659f45574affb021962f285f3f42]) successfully deleted
I20240503 07:59:15.778726 6745 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 07:59:15.778834 6745 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 07:59:15.781491 6745 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 508adb2132b0497195c782a364bb1b40
I20240503 07:59:15.781780 6745 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 4 replicas of tablet b74fd6f87ab44bebb22167d0b6372cd1
I20240503 07:59:15.782059 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 508adb2132b0497195c782a364bb1b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:15.782137 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 508adb2132b0497195c782a364bb1b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:15.782135 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet 508adb2132b0497195c782a364bb1b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:15.782331 9957 tablet_replica.cc:330] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:15.782374 10335 tablet_replica.cc:330] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:15.782501 10335 raft_consensus.cc:2229] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.782514 9957 raft_consensus.cc:2229] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.782636 10335 raft_consensus.cc:2258] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.782651 9957 raft_consensus.cc:2258] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.782858 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet b74fd6f87ab44bebb22167d0b6372cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:15.783169 10335 ts_tablet_manager.cc:1875] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.783170 9957 ts_tablet_manager.cc:1875] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.783362 10947 tablet_replica.cc:330] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:15.783489 10947 raft_consensus.cc:2229] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.783627 10947 raft_consensus.cc:2258] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.783624 9353 tablet_service.cc:1514] Processing DeleteTablet for tablet b74fd6f87ab44bebb22167d0b6372cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:15.783828 6745 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 5c7b63d090d34715b52e64df9701d60d
I20240503 07:59:15.784081 10947 ts_tablet_manager.cc:1875] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.784221 7131 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7b63d090d34715b52e64df9701d60d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:15.784704 11425 tablet_replica.cc:330] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:15.784833 11425 raft_consensus.cc:2229] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.784929 11425 raft_consensus.cc:2258] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.785413 9957 ts_tablet_manager.cc:1888] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:15.785413 10335 ts_tablet_manager.cc:1888] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:15.785426 11425 ts_tablet_manager.cc:1875] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.785477 10335 log.cc:1196] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/508adb2132b0497195c782a364bb1b40
I20240503 07:59:15.785480 9957 log.cc:1196] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/508adb2132b0497195c782a364bb1b40
I20240503 07:59:15.785730 9957 ts_tablet_manager.cc:1909] T 508adb2132b0497195c782a364bb1b40 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:15.785735 10335 ts_tablet_manager.cc:1909] T 508adb2132b0497195c782a364bb1b40 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:15.785907 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet b74fd6f87ab44bebb22167d0b6372cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:15.786399 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet b74fd6f87ab44bebb22167d0b6372cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:15.786542 9957 tablet_replica.cc:330] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:15.786664 9957 raft_consensus.cc:2229] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:15.786677 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 508adb2132b0497195c782a364bb1b40 (table rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]) successfully deleted
I20240503 07:59:15.786890 9957 raft_consensus.cc:2258] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.787364 9957 ts_tablet_manager.cc:1875] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.787372 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7b63d090d34715b52e64df9701d60d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:15.786303 10947 ts_tablet_manager.cc:1888] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.787675 10947 log.cc:1196] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b74fd6f87ab44bebb22167d0b6372cd1
I20240503 07:59:15.787930 11425 ts_tablet_manager.cc:1888] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.787966 10947 ts_tablet_manager.cc:1909] T b74fd6f87ab44bebb22167d0b6372cd1 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:15.787998 11425 log.cc:1196] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/b74fd6f87ab44bebb22167d0b6372cd1
I20240503 07:59:15.788257 11425 ts_tablet_manager.cc:1909] T b74fd6f87ab44bebb22167d0b6372cd1 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:15.788651 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet b74fd6f87ab44bebb22167d0b6372cd1 (table rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]) successfully deleted
I20240503 07:59:15.789147 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet b74fd6f87ab44bebb22167d0b6372cd1 (table rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]) successfully deleted
I20240503 07:59:15.789276 9957 ts_tablet_manager.cc:1888] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.789323 9957 log.cc:1196] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b74fd6f87ab44bebb22167d0b6372cd1
I20240503 07:59:15.789508 9957 ts_tablet_manager.cc:1909] T b74fd6f87ab44bebb22167d0b6372cd1 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:15.789664 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7b63d090d34715b52e64df9701d60d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:15.790311 9957 tablet_replica.cc:330] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:15.790586 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet b74fd6f87ab44bebb22167d0b6372cd1 (table rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]) successfully deleted
I20240503 07:59:15.790752 9957 raft_consensus.cc:2229] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.790938 9957 raft_consensus.cc:2258] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.791414 9957 ts_tablet_manager.cc:1875] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.792650 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 508adb2132b0497195c782a364bb1b40 (table rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]) successfully deleted
I20240503 07:59:15.793598 9957 ts_tablet_manager.cc:1888] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:15.793718 9957 log.cc:1196] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5c7b63d090d34715b52e64df9701d60d
I20240503 07:59:15.793653 10335 tablet_replica.cc:330] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:15.794148 10335 raft_consensus.cc:2229] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.794298 10335 raft_consensus.cc:2258] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.794407 9957 ts_tablet_manager.cc:1909] T 5c7b63d090d34715b52e64df9701d60d P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:15.794857 10335 ts_tablet_manager.cc:1875] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.795054 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 5c7b63d090d34715b52e64df9701d60d (table rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]) successfully deleted
I20240503 07:59:15.797362 10335 ts_tablet_manager.cc:1888] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:15.797493 10335 log.cc:1196] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5c7b63d090d34715b52e64df9701d60d
I20240503 07:59:15.797749 10335 ts_tablet_manager.cc:1909] T 5c7b63d090d34715b52e64df9701d60d P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:15.798388 10335 tablet_replica.cc:330] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:15.798590 11424 tablet_replica.cc:330] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:15.798691 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 5c7b63d090d34715b52e64df9701d60d (table rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]) successfully deleted
I20240503 07:59:15.798836 10335 raft_consensus.cc:2229] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.799058 11424 raft_consensus.cc:2229] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:15.799257 10335 raft_consensus.cc:2258] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.799901 11424 raft_consensus.cc:2258] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.800072 7129 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:15.800243 10335 ts_tablet_manager.cc:1875] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.800734 11424 ts_tablet_manager.cc:1875] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.804538 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:15.804657 10335 ts_tablet_manager.cc:1888] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.804724 10335 log.cc:1196] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b74fd6f87ab44bebb22167d0b6372cd1
I20240503 07:59:15.804983 10335 ts_tablet_manager.cc:1909] T b74fd6f87ab44bebb22167d0b6372cd1 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:15.806492 11424 ts_tablet_manager.cc:1888] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:15.807610 11424 log.cc:1196] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/508adb2132b0497195c782a364bb1b40
I20240503 07:59:15.807982 11424 ts_tablet_manager.cc:1909] T 508adb2132b0497195c782a364bb1b40 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:15.808629 11424 tablet_replica.cc:330] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:15.808634 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet b74fd6f87ab44bebb22167d0b6372cd1 (table rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]) successfully deleted
I20240503 07:59:15.809324 11424 raft_consensus.cc:2229] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:15.809571 11424 raft_consensus.cc:2258] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.810187 11424 ts_tablet_manager.cc:1875] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.810555 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 508adb2132b0497195c782a364bb1b40 (table rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]) successfully deleted
I20240503 07:59:15.811973 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:15.815718 11424 ts_tablet_manager.cc:1888] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:15.816066 11424 log.cc:1196] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5c7b63d090d34715b52e64df9701d60d
I20240503 07:59:15.816740 11424 ts_tablet_manager.cc:1909] T 5c7b63d090d34715b52e64df9701d60d P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:15.818050 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 5c7b63d090d34715b52e64df9701d60d (table rebalance_test_table_18 [id=116b53680e9e4dba8297fef4060bcc55]) successfully deleted
I20240503 07:59:15.826939 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:15.827476 9746 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:15.834820 9484 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:15.841598 6867 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:15.846431 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:15.870749 6745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:15.876714 9614 tablet_service.cc:1467] Processing CreateTablet for tablet 79e6bae6c6d54b2a83702a75cb13f3f1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dcb8debc86c748ffafd7ef49889f3c5d]), partition=
I20240503 07:59:15.876765 9353 tablet_service.cc:1467] Processing CreateTablet for tablet 79e6bae6c6d54b2a83702a75cb13f3f1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dcb8debc86c748ffafd7ef49889f3c5d]), partition=
I20240503 07:59:15.876981 9484 tablet_service.cc:1467] Processing CreateTablet for tablet 79e6bae6c6d54b2a83702a75cb13f3f1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dcb8debc86c748ffafd7ef49889f3c5d]), partition=
I20240503 07:59:15.877043 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e6bae6c6d54b2a83702a75cb13f3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.877043 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e6bae6c6d54b2a83702a75cb13f3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.877259 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e6bae6c6d54b2a83702a75cb13f3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.879145 11200 tablet_bootstrap.cc:492] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:15.879359 11349 tablet_bootstrap.cc:492] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:15.880125 11200 tablet_bootstrap.cc:654] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.880189 11349 tablet_bootstrap.cc:654] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.880894 11204 tablet_bootstrap.cc:492] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:15.881517 11349 tablet_bootstrap.cc:492] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:15.881583 11349 ts_tablet_manager.cc:1367] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:15.881726 11204 tablet_bootstrap.cc:654] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.881925 11349 raft_consensus.cc:348] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.882011 11349 raft_consensus.cc:374] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.882031 11349 raft_consensus.cc:729] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.882130 11349 consensus_queue.cc:260] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.882396 11349 ts_tablet_manager.cc:1398] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.883773 11204 tablet_bootstrap.cc:492] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:15.883869 11204 ts_tablet_manager.cc:1367] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:15.884331 11204 raft_consensus.cc:348] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.884443 11204 raft_consensus.cc:374] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.884472 11204 raft_consensus.cc:729] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.884591 11204 consensus_queue.cc:260] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.884877 11204 ts_tablet_manager.cc:1398] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:15.885648 11200 tablet_bootstrap.cc:492] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:15.885757 11200 ts_tablet_manager.cc:1367] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240503 07:59:15.886206 11200 raft_consensus.cc:348] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.886293 11200 raft_consensus.cc:374] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.886312 11200 raft_consensus.cc:729] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.886444 11200 consensus_queue.cc:260] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.887972 11200 ts_tablet_manager.cc:1398] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:15.903774 7282 consensus_queue.cc:237] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:15.905123 7149 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.905154 9504 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.905541 10486 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.905746 11348 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.907898 11348 raft_consensus.cc:2932] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:15.908290 7149 raft_consensus.cc:2932] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:15.908623 9504 raft_consensus.cc:2932] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:15.910991 6745 catalog_manager.cc:5484] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:15.913938 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d33ff4164b49a7948aa5afee40bf3f with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:15.914196 11424 tablet_replica.cc:330] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:15.914388 11424 raft_consensus.cc:2229] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.914566 11424 raft_consensus.cc:2258] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.915077 11424 ts_tablet_manager.cc:1875] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:15.917486 11424 ts_tablet_manager.cc:1888] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:15.917543 11424 log.cc:1196] T 64d33ff4164b49a7948aa5afee40bf3f P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/64d33ff4164b49a7948aa5afee40bf3f
I20240503 07:59:15.918017 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 64d33ff4164b49a7948aa5afee40bf3f (table rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]) successfully deleted
I20240503 07:59:16.011595 10694 raft_consensus.cc:482] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:16.011720 10694 raft_consensus.cc:504] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.012228 10694 leader_election.cc:288] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:16.012655 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e6bae6c6d54b2a83702a75cb13f3f1" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:16.012821 9504 raft_consensus.cc:2454] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:16.013286 9310 leader_election.cc:302] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:16.013471 10694 raft_consensus.cc:2781] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:16.013547 10694 raft_consensus.cc:482] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:16.013586 10694 raft_consensus.cc:3037] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.014703 10694 raft_consensus.cc:504] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.015102 10694 leader_election.cc:288] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:16.015539 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e6bae6c6d54b2a83702a75cb13f3f1" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:16.015666 9504 raft_consensus.cc:3037] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.016749 9504 raft_consensus.cc:2454] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:16.017063 9310 leader_election.cc:302] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:16.017277 10694 raft_consensus.cc:2781] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:16.017352 10694 raft_consensus.cc:686] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:16.017480 10694 consensus_queue.cc:237] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.019230 6746 catalog_manager.cc:5484] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:16.020764 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e6bae6c6d54b2a83702a75cb13f3f1" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:16.020906 9635 raft_consensus.cc:2454] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:16.021148 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e6bae6c6d54b2a83702a75cb13f3f1" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:16.021260 9636 raft_consensus.cc:3037] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.022085 9636 raft_consensus.cc:2454] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:16.041491 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:16.044073 9353 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:16.046042 7129 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:16.055099 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:16.060792 9484 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:16.069803 9746 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:16.072373 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:16.073532 6867 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:16.087837 11299 consensus_queue.cc:1035] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.093415 11299 consensus_queue.cc:1035] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.099624 10651 consensus_queue.cc:1035] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.106148 11347 consensus_queue.cc:1035] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.131644 6886 consensus_queue.cc:237] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:16.133090 7020 raft_consensus.cc:1261] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.133292 9767 raft_consensus.cc:1261] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.133657 10651 consensus_queue.cc:1035] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.133891 11347 consensus_queue.cc:1035] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.136276 11347 raft_consensus.cc:2932] T eef64b6b6db54db2b47314d672269043 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:16.136907 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:16.137588 7020 raft_consensus.cc:2932] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:16.137995 9767 raft_consensus.cc:2932] T eef64b6b6db54db2b47314d672269043 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:16.141793 6742 catalog_manager.cc:5484] T eef64b6b6db54db2b47314d672269043 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:16.143587 9636 raft_consensus.cc:1261] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:16.143764 9504 raft_consensus.cc:1261] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:16.144294 10931 consensus_queue.cc:1035] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.145112 10931 consensus_queue.cc:1035] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.147173 6739 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 79e6bae6c6d54b2a83702a75cb13f3f1 (table rebalancer_dynamic_table [id=dcb8debc86c748ffafd7ef49889f3c5d]). Expected version 1 got 0
I20240503 07:59:16.149755 6739 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 79e6bae6c6d54b2a83702a75cb13f3f1 (table rebalancer_dynamic_table [id=dcb8debc86c748ffafd7ef49889f3c5d]). Expected version 1 got 0
I20240503 07:59:16.150427 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet eef64b6b6db54db2b47314d672269043 with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:16.150733 10947 tablet_replica.cc:330] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:16.150887 10947 raft_consensus.cc:2229] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.151063 10947 raft_consensus.cc:2258] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.151613 10947 ts_tablet_manager.cc:1875] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.151963 11300 tablet.cc:1722] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:16.152298 11300 tablet.cc:1620] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:16.153317 11296 tablet.cc:1722] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:16.153666 11296 tablet.cc:1620] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:16.153926 10947 ts_tablet_manager.cc:1888] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:16.153991 10947 log.cc:1196] T eef64b6b6db54db2b47314d672269043 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/eef64b6b6db54db2b47314d672269043
I20240503 07:59:16.155200 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet eef64b6b6db54db2b47314d672269043 (table rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]) successfully deleted
I20240503 07:59:16.157341 11542 tablet.cc:1722] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:16.157533 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=dcb8debc86c748ffafd7ef49889f3c5d] alter complete (version 1)
I20240503 07:59:16.158531 11542 tablet.cc:1620] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:16.241175 7129 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:16.244284 9353 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:16.244957 6867 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:16.255303 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:16.258935 9746 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:16.259375 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:16.261687 6742 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:16.275936 11542 tablet.cc:1722] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:16.276340 11542 tablet.cc:1620] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:16.278174 11300 tablet.cc:1722] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:16.278538 11300 tablet.cc:1620] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:16.284025 11296 tablet.cc:1722] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:16.284385 11296 tablet.cc:1620] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:16.285617 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:16.287786 6742 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=dcb8debc86c748ffafd7ef49889f3c5d] alter complete (version 2)
I20240503 07:59:16.292207 9484 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:16.295938 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:16.304632 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 6e051a416bb446df91e90eba13206341 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=4b3fe402f6344742a8841ffd09a9f966]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.304991 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e051a416bb446df91e90eba13206341. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.305136 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 5a838dc6e3114be5aceac51d567bc645 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=4b3fe402f6344742a8841ffd09a9f966]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.305404 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a838dc6e3114be5aceac51d567bc645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.306694 9614 tablet_service.cc:1467] Processing CreateTablet for tablet 5a838dc6e3114be5aceac51d567bc645 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=4b3fe402f6344742a8841ffd09a9f966]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.306958 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a838dc6e3114be5aceac51d567bc645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.307639 11589 tablet_bootstrap.cc:492] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:16.308578 11589 tablet_bootstrap.cc:654] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.308595 9484 tablet_service.cc:1467] Processing CreateTablet for tablet 6e051a416bb446df91e90eba13206341 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=4b3fe402f6344742a8841ffd09a9f966]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.308873 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e051a416bb446df91e90eba13206341. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.309484 11204 tablet_bootstrap.cc:492] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:16.309479 9353 tablet_service.cc:1467] Processing CreateTablet for tablet 6e051a416bb446df91e90eba13206341 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=4b3fe402f6344742a8841ffd09a9f966]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.309731 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e051a416bb446df91e90eba13206341. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.310269 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 40846ecb9f5d4c39a7bafac29c5d4e8a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=4b3fe402f6344742a8841ffd09a9f966]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.310550 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40846ecb9f5d4c39a7bafac29c5d4e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.310606 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 5a838dc6e3114be5aceac51d567bc645 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=4b3fe402f6344742a8841ffd09a9f966]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.310801 11204 tablet_bootstrap.cc:654] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.310843 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a838dc6e3114be5aceac51d567bc645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.312024 9614 tablet_service.cc:1467] Processing CreateTablet for tablet 40846ecb9f5d4c39a7bafac29c5d4e8a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=4b3fe402f6344742a8841ffd09a9f966]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.312270 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40846ecb9f5d4c39a7bafac29c5d4e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.312673 11349 tablet_bootstrap.cc:492] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:16.312673 11200 tablet_bootstrap.cc:492] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:16.313491 11200 tablet_bootstrap.cc:654] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.313491 11349 tablet_bootstrap.cc:654] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.315666 11422 consensus_queue.cc:1035] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.316870 11204 tablet_bootstrap.cc:492] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:16.316962 11204 ts_tablet_manager.cc:1367] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 07:59:16.317437 11204 raft_consensus.cc:348] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.317538 11204 raft_consensus.cc:374] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.317564 11204 raft_consensus.cc:729] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.317672 11204 consensus_queue.cc:260] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.317945 11204 ts_tablet_manager.cc:1398] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:16.318123 11204 tablet_bootstrap.cc:492] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:16.319826 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 40846ecb9f5d4c39a7bafac29c5d4e8a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=4b3fe402f6344742a8841ffd09a9f966]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.320148 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40846ecb9f5d4c39a7bafac29c5d4e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.321192 11204 tablet_bootstrap.cc:654] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.323789 11204 tablet_bootstrap.cc:492] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:16.323890 11204 ts_tablet_manager.cc:1367] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:16.324374 11204 raft_consensus.cc:348] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:16.324524 11204 raft_consensus.cc:374] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.324569 11204 raft_consensus.cc:729] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.324725 11204 consensus_queue.cc:260] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:16.325079 11204 ts_tablet_manager.cc:1398] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.326912 11588 tablet_bootstrap.cc:492] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:16.329366 11588 tablet_bootstrap.cc:654] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.330523 11349 tablet_bootstrap.cc:492] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:16.330626 11349 ts_tablet_manager.cc:1367] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.000s
I20240503 07:59:16.330729 11589 tablet_bootstrap.cc:492] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:16.330839 11589 ts_tablet_manager.cc:1367] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240503 07:59:16.331113 11349 raft_consensus.cc:348] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:16.331238 11349 raft_consensus.cc:374] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.331269 11349 raft_consensus.cc:729] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.331306 11589 raft_consensus.cc:348] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.331411 11589 raft_consensus.cc:374] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.331437 11589 raft_consensus.cc:729] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.331401 11349 consensus_queue.cc:260] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:16.331549 11589 consensus_queue.cc:260] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.331691 11349 ts_tablet_manager.cc:1398] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:16.331804 11589 ts_tablet_manager.cc:1398] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:16.331837 11349 tablet_bootstrap.cc:492] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:16.332233 11588 tablet_bootstrap.cc:492] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:16.332324 11588 ts_tablet_manager.cc:1367] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:16.332783 11349 tablet_bootstrap.cc:654] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.335397 11349 tablet_bootstrap.cc:492] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:16.335556 11349 ts_tablet_manager.cc:1367] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 07:59:16.335526 11588 raft_consensus.cc:348] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.335719 11588 raft_consensus.cc:374] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.335805 11588 raft_consensus.cc:729] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.335995 11588 consensus_queue.cc:260] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.336177 11349 raft_consensus.cc:348] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.336396 11349 raft_consensus.cc:374] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.336503 11588 ts_tablet_manager.cc:1398] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:16.336520 11349 raft_consensus.cc:729] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.336640 11349 consensus_queue.cc:260] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.336741 11588 tablet_bootstrap.cc:492] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:16.337122 11349 ts_tablet_manager.cc:1398] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.337507 11200 tablet_bootstrap.cc:492] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:16.337610 11200 ts_tablet_manager.cc:1367] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 07:59:16.338035 11588 tablet_bootstrap.cc:654] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.338076 11200 raft_consensus.cc:348] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.338178 11200 raft_consensus.cc:374] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.338209 11200 raft_consensus.cc:729] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.338326 11200 consensus_queue.cc:260] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.338618 11200 ts_tablet_manager.cc:1398] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.338816 11200 tablet_bootstrap.cc:492] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:16.339185 11588 tablet_bootstrap.cc:492] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:16.339268 11588 ts_tablet_manager.cc:1367] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:16.339767 11588 raft_consensus.cc:348] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:16.339859 11200 tablet_bootstrap.cc:654] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.339874 11588 raft_consensus.cc:374] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.339907 11588 raft_consensus.cc:729] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.340035 11588 consensus_queue.cc:260] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:16.340337 11588 ts_tablet_manager.cc:1398] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:16.341564 10716 consensus_queue.cc:1035] T da9b313c416a49239cb266f90445ec4a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.345928 11200 tablet_bootstrap.cc:492] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:16.346042 11200 ts_tablet_manager.cc:1367] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:16.346551 11200 raft_consensus.cc:348] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.346653 11200 raft_consensus.cc:374] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.346681 11200 raft_consensus.cc:729] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.346789 11200 consensus_queue.cc:260] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.347050 11200 ts_tablet_manager.cc:1398] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.376991 7282 consensus_queue.cc:237] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.378582 9504 raft_consensus.cc:1261] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.378661 7150 raft_consensus.cc:1261] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.379082 11348 consensus_queue.cc:1035] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.379292 10486 consensus_queue.cc:1035] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.381312 10486 raft_consensus.cc:2932] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:16.381755 9505 raft_consensus.cc:2932] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:16.384379 6742 catalog_manager.cc:5484] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:16.384652 7150 raft_consensus.cc:2932] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:16.387243 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet dd6aa70d9c4c4baea597bbb9788fdb37 with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:16.387490 11424 tablet_replica.cc:330] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:16.387632 11424 raft_consensus.cc:2229] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.387791 11424 raft_consensus.cc:2258] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.388361 11424 ts_tablet_manager.cc:1875] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.390563 11424 ts_tablet_manager.cc:1888] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:16.390630 11424 log.cc:1196] T dd6aa70d9c4c4baea597bbb9788fdb37 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/dd6aa70d9c4c4baea597bbb9788fdb37
I20240503 07:59:16.391095 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet dd6aa70d9c4c4baea597bbb9788fdb37 (table rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]) successfully deleted
I20240503 07:59:16.397459 6742 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:16.403949 11542 tablet.cc:1722] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:16.405550 11296 tablet.cc:1722] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:16.413172 6742 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=dcb8debc86c748ffafd7ef49889f3c5d] alter complete (version 3)
I20240503 07:59:16.413867 11300 tablet.cc:1722] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:16.434455 11422 raft_consensus.cc:482] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:16.434579 11422 raft_consensus.cc:504] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.435030 11422 leader_election.cc:288] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:16.435365 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a838dc6e3114be5aceac51d567bc645" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:16.435499 9505 raft_consensus.cc:2454] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:16.435551 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a838dc6e3114be5aceac51d567bc645" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:16.435700 9636 raft_consensus.cc:2454] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:16.435782 7087 leader_election.cc:302] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:16.435941 11422 raft_consensus.cc:2781] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:16.436004 11422 raft_consensus.cc:482] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:16.436036 11422 raft_consensus.cc:3037] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.437129 11422 raft_consensus.cc:504] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.437467 11422 leader_election.cc:288] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:16.437942 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a838dc6e3114be5aceac51d567bc645" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:16.438056 9636 raft_consensus.cc:3037] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.438102 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a838dc6e3114be5aceac51d567bc645" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:16.438225 9505 raft_consensus.cc:3037] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.439056 9636 raft_consensus.cc:2454] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:16.439354 9505 raft_consensus.cc:2454] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:16.439447 7084 leader_election.cc:302] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2; no voters:
I20240503 07:59:16.439771 11422 raft_consensus.cc:2781] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:16.439855 11422 raft_consensus.cc:686] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:16.439997 11422 consensus_queue.cc:237] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.442095 6739 catalog_manager.cc:5484] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:16.481654 11299 raft_consensus.cc:482] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:16.481849 11299 raft_consensus.cc:504] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.482367 11299 leader_election.cc:288] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:16.482758 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e051a416bb446df91e90eba13206341" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:16.482913 9357 raft_consensus.cc:2454] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:16.483219 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e051a416bb446df91e90eba13206341" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:16.483273 9439 leader_election.cc:302] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:16.483366 9767 raft_consensus.cc:2454] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:16.483448 11299 raft_consensus.cc:2781] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:16.483510 11299 raft_consensus.cc:482] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:16.483543 11299 raft_consensus.cc:3037] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.484624 11299 raft_consensus.cc:504] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.484992 11299 leader_election.cc:288] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:16.485332 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e051a416bb446df91e90eba13206341" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:16.485440 9767 raft_consensus.cc:3037] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.486210 9767 raft_consensus.cc:2454] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:16.486493 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e051a416bb446df91e90eba13206341" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:16.486555 9441 leader_election.cc:302] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:16.486635 9357 raft_consensus.cc:3037] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.486809 11299 raft_consensus.cc:2781] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:16.486918 11299 raft_consensus.cc:686] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:16.487061 11299 consensus_queue.cc:237] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:16.487680 9357 raft_consensus.cc:2454] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:16.489005 6744 catalog_manager.cc:5484] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:16.502993 11343 raft_consensus.cc:482] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:16.503118 11343 raft_consensus.cc:504] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:16.503582 11343 leader_election.cc:288] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:16.503937 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40846ecb9f5d4c39a7bafac29c5d4e8a" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:16.504052 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40846ecb9f5d4c39a7bafac29c5d4e8a" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:16.504122 9357 raft_consensus.cc:2454] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:16.504185 9636 raft_consensus.cc:2454] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:16.504475 9701 leader_election.cc:302] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:16.504659 11343 raft_consensus.cc:2781] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:16.504724 11343 raft_consensus.cc:482] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:16.504750 11343 raft_consensus.cc:3037] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.505764 11343 raft_consensus.cc:504] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:16.506227 11343 leader_election.cc:288] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:16.506832 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40846ecb9f5d4c39a7bafac29c5d4e8a" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:16.506974 9357 raft_consensus.cc:3037] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.507473 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40846ecb9f5d4c39a7bafac29c5d4e8a" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:16.507598 9636 raft_consensus.cc:3037] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.508042 9357 raft_consensus.cc:2454] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:16.508322 9701 leader_election.cc:302] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:16.508567 11343 raft_consensus.cc:2781] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:16.508600 9636 raft_consensus.cc:2454] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:16.508801 11343 raft_consensus.cc:686] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:16.508960 11343 consensus_queue.cc:237] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:16.510908 6743 catalog_manager.cc:5484] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:16.518364 6743 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:16.518553 6743 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:16.521581 6743 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 79e6bae6c6d54b2a83702a75cb13f3f1
I20240503 07:59:16.523336 9614 tablet_service.cc:1514] Processing DeleteTablet for tablet 79e6bae6c6d54b2a83702a75cb13f3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:16 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:16.523628 9484 tablet_service.cc:1514] Processing DeleteTablet for tablet 79e6bae6c6d54b2a83702a75cb13f3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:16 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:16.523999 9353 tablet_service.cc:1514] Processing DeleteTablet for tablet 79e6bae6c6d54b2a83702a75cb13f3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:16 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:16.524051 11676 tablet_replica.cc:330] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:16.524207 11676 raft_consensus.cc:2229] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.524305 11677 tablet_replica.cc:330] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:16.525379 11676 raft_consensus.cc:2258] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.525668 11677 raft_consensus.cc:2229] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:16.525964 11677 raft_consensus.cc:2258] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.527778 11677 ts_tablet_manager.cc:1875] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:16.528633 11676 ts_tablet_manager.cc:1875] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:16.531081 6867 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:16.532243 11677 ts_tablet_manager.cc:1888] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:16.534205 11676 ts_tablet_manager.cc:1888] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:16.534292 11676 log.cc:1196] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/79e6bae6c6d54b2a83702a75cb13f3f1
I20240503 07:59:16.534842 9746 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:16.535804 11677 log.cc:1196] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/79e6bae6c6d54b2a83702a75cb13f3f1
I20240503 07:59:16.536055 7129 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:16.536374 11676 ts_tablet_manager.cc:1909] T 79e6bae6c6d54b2a83702a75cb13f3f1 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:16.536533 11677 ts_tablet_manager.cc:1909] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:16.538525 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:16.539193 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 79e6bae6c6d54b2a83702a75cb13f3f1 (table rebalancer_dynamic_table_new_name [id=dcb8debc86c748ffafd7ef49889f3c5d]) successfully deleted
I20240503 07:59:16.539680 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 79e6bae6c6d54b2a83702a75cb13f3f1 (table rebalancer_dynamic_table_new_name [id=dcb8debc86c748ffafd7ef49889f3c5d]) successfully deleted
I20240503 07:59:16.556128 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:16.565448 11675 tablet_replica.cc:330] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:16.565704 11675 raft_consensus.cc:2229] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.565888 11675 raft_consensus.cc:2258] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.566699 11675 ts_tablet_manager.cc:1875] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:16.568866 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:16.569208 11675 ts_tablet_manager.cc:1888] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:16.569348 11675 log.cc:1196] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/79e6bae6c6d54b2a83702a75cb13f3f1
I20240503 07:59:16.569643 11675 ts_tablet_manager.cc:1909] T 79e6bae6c6d54b2a83702a75cb13f3f1 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:16.570706 9484 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:16.572153 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 79e6bae6c6d54b2a83702a75cb13f3f1 (table rebalancer_dynamic_table_new_name [id=dcb8debc86c748ffafd7ef49889f3c5d]) successfully deleted
I20240503 07:59:16.572667 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:16.624889 6744 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:16.630789 9484 tablet_service.cc:1467] Processing CreateTablet for tablet 22a5686856da4fc6b1d9707837a16922 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5bf71ba7bc3e45c48bbaee1c5f0f9098]), partition=
I20240503 07:59:16.630888 9614 tablet_service.cc:1467] Processing CreateTablet for tablet 22a5686856da4fc6b1d9707837a16922 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5bf71ba7bc3e45c48bbaee1c5f0f9098]), partition=
I20240503 07:59:16.631150 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a5686856da4fc6b1d9707837a16922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.631124 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a5686856da4fc6b1d9707837a16922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.633041 11200 tablet_bootstrap.cc:492] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:16.633383 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 22a5686856da4fc6b1d9707837a16922 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5bf71ba7bc3e45c48bbaee1c5f0f9098]), partition=
I20240503 07:59:16.633620 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a5686856da4fc6b1d9707837a16922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.634275 11200 tablet_bootstrap.cc:654] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.635478 11200 tablet_bootstrap.cc:492] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:16.635560 11200 ts_tablet_manager.cc:1367] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:16.635869 11589 tablet_bootstrap.cc:492] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:16.635954 11204 tablet_bootstrap.cc:492] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:16.636039 11200 raft_consensus.cc:348] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.636139 11200 raft_consensus.cc:374] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.636168 11200 raft_consensus.cc:729] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.636287 11200 consensus_queue.cc:260] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.636545 11200 ts_tablet_manager.cc:1398] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.636822 11589 tablet_bootstrap.cc:654] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.636989 11204 tablet_bootstrap.cc:654] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.638096 11204 tablet_bootstrap.cc:492] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:16.638096 11589 tablet_bootstrap.cc:492] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:16.638187 11204 ts_tablet_manager.cc:1367] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:16.638190 11589 ts_tablet_manager.cc:1367] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:16.638661 11204 raft_consensus.cc:348] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.638671 11589 raft_consensus.cc:348] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.638767 11589 raft_consensus.cc:374] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.638767 11204 raft_consensus.cc:374] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.638796 11204 raft_consensus.cc:729] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.638796 11589 raft_consensus.cc:729] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.638912 11204 consensus_queue.cc:260] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.638912 11589 consensus_queue.cc:260] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.639165 11204 ts_tablet_manager.cc:1398] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.639176 11589 ts_tablet_manager.cc:1398] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.642508 7150 consensus_queue.cc:237] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:16.643958 9767 raft_consensus.cc:1261] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.644037 7020 raft_consensus.cc:1261] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.644408 11422 consensus_queue.cc:1035] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.644717 11422 consensus_queue.cc:1035] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.647172 11699 raft_consensus.cc:2932] T 921bd35f61ed4465aef45eee3383c9b3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:16.647557 9767 raft_consensus.cc:2932] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:16.647636 7020 raft_consensus.cc:2932] T 921bd35f61ed4465aef45eee3383c9b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:16.650197 6744 catalog_manager.cc:5484] T 921bd35f61ed4465aef45eee3383c9b3 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:16.655639 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 921bd35f61ed4465aef45eee3383c9b3 with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:16.655984 11705 tablet_replica.cc:330] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:16.656143 11705 raft_consensus.cc:2229] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.656296 11705 raft_consensus.cc:2258] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.656811 11705 ts_tablet_manager.cc:1875] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.659286 11705 ts_tablet_manager.cc:1888] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:16.659363 11705 log.cc:1196] T 921bd35f61ed4465aef45eee3383c9b3 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/921bd35f61ed4465aef45eee3383c9b3
I20240503 07:59:16.659855 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 921bd35f61ed4465aef45eee3383c9b3 (table rebalance_test_table_11 [id=9caa58273688477ab18820b0590675ab]) successfully deleted
I20240503 07:59:16.788763 11299 raft_consensus.cc:482] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:16.788903 11299 raft_consensus.cc:504] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.789438 11299 leader_election.cc:288] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:16.789875 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a5686856da4fc6b1d9707837a16922" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:16.789885 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a5686856da4fc6b1d9707837a16922" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:16.790035 7150 raft_consensus.cc:2454] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:16.790035 9636 raft_consensus.cc:2454] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:16.790321 9438 leader_election.cc:302] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:16.790534 11299 raft_consensus.cc:2781] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:16.790587 11299 raft_consensus.cc:482] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:16.790621 11299 raft_consensus.cc:3037] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.791620 11299 raft_consensus.cc:504] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.791955 11299 leader_election.cc:288] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:16.792276 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a5686856da4fc6b1d9707837a16922" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:16.792306 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a5686856da4fc6b1d9707837a16922" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:16.792408 9636 raft_consensus.cc:3037] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.792421 7150 raft_consensus.cc:3037] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.793174 9636 raft_consensus.cc:2454] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:16.793200 7150 raft_consensus.cc:2454] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:16.793494 9441 leader_election.cc:302] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:16.793651 11299 raft_consensus.cc:2781] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:16.793730 11299 raft_consensus.cc:686] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:16.793879 11299 consensus_queue.cc:237] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:16.795773 6744 catalog_manager.cc:5484] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:16.816474 7129 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:16.821357 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:16.822151 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:16.823482 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:16.827893 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:16.846549 6867 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:16.847558 9746 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:16.853305 9484 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:16.878319 11299 consensus_queue.cc:1035] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.881708 11299 consensus_queue.cc:1035] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.905391 7149 consensus_queue.cc:237] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:16.906792 7282 raft_consensus.cc:1261] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.906872 9767 raft_consensus.cc:1261] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.907223 11699 consensus_queue.cc:1035] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.907457 10716 consensus_queue.cc:1035] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.909493 11422 raft_consensus.cc:2932] T 7e8e04c50d7147af903d7113df3d9744 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:16.910014 7282 raft_consensus.cc:2932] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:16.910246 9767 raft_consensus.cc:2932] T 7e8e04c50d7147af903d7113df3d9744 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:16.912976 6745 catalog_manager.cc:5484] T 7e8e04c50d7147af903d7113df3d9744 P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:16.916237 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e8e04c50d7147af903d7113df3d9744 with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:16.916997 11779 tablet_replica.cc:330] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:16.917179 11779 raft_consensus.cc:2229] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.917824 11779 raft_consensus.cc:2258] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.918471 11779 ts_tablet_manager.cc:1875] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.920796 11779 ts_tablet_manager.cc:1888] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:16.920886 11779 log.cc:1196] T 7e8e04c50d7147af903d7113df3d9744 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7e8e04c50d7147af903d7113df3d9744
I20240503 07:59:16.921408 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 7e8e04c50d7147af903d7113df3d9744 (table rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]) successfully deleted
I20240503 07:59:16.936270 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:16.941087 9636 raft_consensus.cc:1261] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:16.941087 7149 raft_consensus.cc:1261] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:16.941815 10379 consensus_queue.cc:1035] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.942014 11299 consensus_queue.cc:1035] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.944762 6741 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 22a5686856da4fc6b1d9707837a16922 (table rebalancer_dynamic_table [id=5bf71ba7bc3e45c48bbaee1c5f0f9098]). Expected version 1 got 0
I20240503 07:59:16.946008 6741 catalog_manager.cc:5576] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317) does not have the latest schema for tablet 22a5686856da4fc6b1d9707837a16922 (table rebalancer_dynamic_table [id=5bf71ba7bc3e45c48bbaee1c5f0f9098]). Expected version 1 got 0
I20240503 07:59:16.946909 11790 tablet.cc:1722] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:16.946168 11789 tablet.cc:1722] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:16.947273 11790 tablet.cc:1620] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:16.947394 11789 tablet.cc:1620] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:16.949369 6744 catalog_manager.cc:5576] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317) does not have the latest schema for tablet 22a5686856da4fc6b1d9707837a16922 (table rebalancer_dynamic_table [id=5bf71ba7bc3e45c48bbaee1c5f0f9098]). Expected version 1 got 0
I20240503 07:59:16.950582 6741 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 22a5686856da4fc6b1d9707837a16922 (table rebalancer_dynamic_table [id=5bf71ba7bc3e45c48bbaee1c5f0f9098]). Expected version 1 got 0
I20240503 07:59:16.952642 11795 tablet.cc:1722] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:16.953048 11795 tablet.cc:1620] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:16.953128 6744 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=5bf71ba7bc3e45c48bbaee1c5f0f9098] alter complete (version 1)
I20240503 07:59:16.967352 11343 consensus_queue.cc:1035] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.970994 11662 consensus_queue.cc:1035] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.979305 11422 consensus_queue.cc:1035] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.983268 11699 consensus_queue.cc:1035] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.024828 7129 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:17.028059 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.039237 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.040050 9746 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.040937 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:17.041074 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:17.048099 6745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:17.054692 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:17.060194 9484 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:17.062508 11789 tablet.cc:1722] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:17.062875 11789 tablet.cc:1620] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:17.063587 11790 tablet.cc:1722] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:17.063872 11790 tablet.cc:1620] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:17.064965 11795 tablet.cc:1722] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:17.065258 11795 tablet.cc:1620] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:17.067925 9746 tablet_service.cc:1467] Processing CreateTablet for tablet b3503f0e1e29430a933986978978f1d4 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f0fcbc958ed14af18b82b7bb244d2bfb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.068043 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 23de5c1fe8c44b3c990f1128826a72b0 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f0fcbc958ed14af18b82b7bb244d2bfb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.068224 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3503f0e1e29430a933986978978f1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.068179 6867 tablet_service.cc:1467] Processing CreateTablet for tablet b3503f0e1e29430a933986978978f1d4 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f0fcbc958ed14af18b82b7bb244d2bfb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.068446 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3503f0e1e29430a933986978978f1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.069000 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23de5c1fe8c44b3c990f1128826a72b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.069167 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 5388c1fa25bc4f19a2aef9d8960e88aa (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f0fcbc958ed14af18b82b7bb244d2bfb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.069378 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5388c1fa25bc4f19a2aef9d8960e88aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.071229 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 23de5c1fe8c44b3c990f1128826a72b0 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f0fcbc958ed14af18b82b7bb244d2bfb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.071532 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23de5c1fe8c44b3c990f1128826a72b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.072078 9484 tablet_service.cc:1467] Processing CreateTablet for tablet 23de5c1fe8c44b3c990f1128826a72b0 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f0fcbc958ed14af18b82b7bb244d2bfb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.072365 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23de5c1fe8c44b3c990f1128826a72b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.074162 9614 tablet_service.cc:1467] Processing CreateTablet for tablet b3503f0e1e29430a933986978978f1d4 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f0fcbc958ed14af18b82b7bb244d2bfb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.074584 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 5388c1fa25bc4f19a2aef9d8960e88aa (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f0fcbc958ed14af18b82b7bb244d2bfb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.074805 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5388c1fa25bc4f19a2aef9d8960e88aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.075637 11832 tablet_bootstrap.cc:492] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:17.076958 6868 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:17.077209 11200 tablet_bootstrap.cc:492] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:17.077059 11832 tablet_bootstrap.cc:654] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.078085 11200 tablet_bootstrap.cc:654] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.078960 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3503f0e1e29430a933986978978f1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.081281 11204 tablet_bootstrap.cc:492] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:17.082238 11204 tablet_bootstrap.cc:654] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.083966 6742 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=5bf71ba7bc3e45c48bbaee1c5f0f9098] alter complete (version 2)
I20240503 07:59:17.099263 11831 tablet_bootstrap.cc:492] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:17.100503 11831 tablet_bootstrap.cc:654] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.101778 11200 tablet_bootstrap.cc:492] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:17.101905 11200 ts_tablet_manager.cc:1367] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.000s
I20240503 07:59:17.102474 11200 raft_consensus.cc:348] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:17.102636 11200 raft_consensus.cc:374] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.102676 11200 raft_consensus.cc:729] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.102804 11200 consensus_queue.cc:260] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:17.102876 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 5388c1fa25bc4f19a2aef9d8960e88aa (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f0fcbc958ed14af18b82b7bb244d2bfb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.103098 11200 ts_tablet_manager.cc:1398] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:17.103206 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5388c1fa25bc4f19a2aef9d8960e88aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.103296 11200 tablet_bootstrap.cc:492] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:17.104897 11200 tablet_bootstrap.cc:654] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.107194 11200 tablet_bootstrap.cc:492] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:17.107503 11200 ts_tablet_manager.cc:1367] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:17.107942 11200 raft_consensus.cc:348] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.108050 11200 raft_consensus.cc:374] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.108080 11200 raft_consensus.cc:729] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.108238 11200 consensus_queue.cc:260] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.109032 11200 ts_tablet_manager.cc:1398] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.111580 11831 tablet_bootstrap.cc:492] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:17.111768 11831 ts_tablet_manager.cc:1367] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:17.112632 11831 raft_consensus.cc:348] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.112891 11831 raft_consensus.cc:374] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.112965 11831 raft_consensus.cc:729] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.113173 11831 consensus_queue.cc:260] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.113561 11831 ts_tablet_manager.cc:1398] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:17.114696 11832 tablet_bootstrap.cc:492] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:17.114835 11832 ts_tablet_manager.cc:1367] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20240503 07:59:17.115368 11832 raft_consensus.cc:348] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:17.115484 11832 raft_consensus.cc:374] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.115512 11832 raft_consensus.cc:729] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.115636 11832 consensus_queue.cc:260] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:17.115933 11832 ts_tablet_manager.cc:1398] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.116115 11832 tablet_bootstrap.cc:492] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:17.117169 11832 tablet_bootstrap.cc:654] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.118767 11832 tablet_bootstrap.cc:492] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:17.118927 11832 ts_tablet_manager.cc:1367] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:17.119372 11832 raft_consensus.cc:348] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.119551 11832 raft_consensus.cc:374] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.119627 11832 raft_consensus.cc:729] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.119797 11832 consensus_queue.cc:260] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.120193 11832 ts_tablet_manager.cc:1398] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.120465 11832 tablet_bootstrap.cc:492] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:17.121384 11832 tablet_bootstrap.cc:654] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.123314 11832 tablet_bootstrap.cc:492] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:17.123400 11832 ts_tablet_manager.cc:1367] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:17.123872 11832 raft_consensus.cc:348] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:17.123965 11832 raft_consensus.cc:374] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.123993 11832 raft_consensus.cc:729] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.124130 11832 consensus_queue.cc:260] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:17.124487 11832 ts_tablet_manager.cc:1398] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.127178 11204 tablet_bootstrap.cc:492] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:17.127274 11204 ts_tablet_manager.cc:1367] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20240503 07:59:17.127713 11204 raft_consensus.cc:348] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:17.127825 11204 raft_consensus.cc:374] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.127852 11204 raft_consensus.cc:729] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.127976 11204 consensus_queue.cc:260] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:17.128123 11830 tablet_bootstrap.cc:492] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:17.128252 11204 ts_tablet_manager.cc:1398] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.129123 11830 tablet_bootstrap.cc:654] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.132220 11830 tablet_bootstrap.cc:492] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:17.132320 11830 ts_tablet_manager.cc:1367] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:17.132882 11830 raft_consensus.cc:348] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:17.133024 11830 raft_consensus.cc:374] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.133121 11830 raft_consensus.cc:729] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.133311 11830 consensus_queue.cc:260] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:17.133863 11830 ts_tablet_manager.cc:1398] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:17.134055 11830 tablet_bootstrap.cc:492] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:17.135182 11830 tablet_bootstrap.cc:654] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.136303 11830 tablet_bootstrap.cc:492] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:17.136447 11830 ts_tablet_manager.cc:1367] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:17.136966 11830 raft_consensus.cc:348] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:17.137143 11830 raft_consensus.cc:374] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.137215 11830 raft_consensus.cc:729] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.137395 11830 consensus_queue.cc:260] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:17.137809 11830 ts_tablet_manager.cc:1398] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.147961 11299 raft_consensus.cc:482] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.148104 11299 raft_consensus.cc:504] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:17.148545 11299 leader_election.cc:288] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:17.149129 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5388c1fa25bc4f19a2aef9d8960e88aa" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:17.149278 6886 raft_consensus.cc:2454] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:17.149559 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5388c1fa25bc4f19a2aef9d8960e88aa" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:17.149605 9438 leader_election.cc:302] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:17.149713 9767 raft_consensus.cc:2454] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:17.149791 11299 raft_consensus.cc:2781] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:17.149856 11299 raft_consensus.cc:482] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:17.149891 11299 raft_consensus.cc:3037] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.151077 11299 raft_consensus.cc:504] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:17.151450 11299 leader_election.cc:288] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:17.152240 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5388c1fa25bc4f19a2aef9d8960e88aa" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:17.152365 6886 raft_consensus.cc:3037] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.152320 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5388c1fa25bc4f19a2aef9d8960e88aa" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:17.152452 9767 raft_consensus.cc:3037] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.153326 9767 raft_consensus.cc:2454] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:17.153412 6886 raft_consensus.cc:2454] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:17.153661 9441 leader_election.cc:302] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:17.153875 11299 raft_consensus.cc:2781] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:17.153983 11299 raft_consensus.cc:686] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:17.154142 11299 consensus_queue.cc:237] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:17.156080 6742 catalog_manager.cc:5484] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:17.178158 7020 consensus_queue.cc:237] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:17.179895 7149 raft_consensus.cc:1261] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:17.180182 7278 raft_consensus.cc:1261] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:17.180371 11596 consensus_queue.cc:1035] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:17.180898 11596 consensus_queue.cc:1035] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:17.185261 11859 raft_consensus.cc:2932] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
W20240503 07:59:17.185802 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abb55c09be324f37a1a546e73008a80d. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:17.186146 7278 raft_consensus.cc:2932] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:17.188972 6742 catalog_manager.cc:5484] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:17.190971 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:17.192595 7149 raft_consensus.cc:2932] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:17.199750 11789 tablet.cc:1722] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:17.200158 11790 tablet.cc:1722] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:17.204384 11795 tablet.cc:1722] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:17.204785 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=5bf71ba7bc3e45c48bbaee1c5f0f9098] alter complete (version 3)
I20240503 07:59:17.224437 9767 consensus_queue.cc:237] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:17.226068 9357 raft_consensus.cc:1261] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.226537 9636 raft_consensus.cc:1261] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:17.226521 9700 consensus_peers.cc:485] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a -> Peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Couldn't send request to peer c4218904dd7e42d2be95b901a6e5db92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40846ecb9f5d4c39a7bafac29c5d4e8a. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:17.226802 11879 consensus_queue.cc:1035] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:17.226959 11662 consensus_queue.cc:1035] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:17.229493 11662 raft_consensus.cc:2932] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:17.230999 9636 raft_consensus.cc:2932] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:17.231819 9357 raft_consensus.cc:2932] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:17.233806 6741 catalog_manager.cc:5484] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:17.258299 9505 consensus_queue.cc:237] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:17.259845 7020 raft_consensus.cc:1261] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.259908 9636 raft_consensus.cc:1261] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.260238 10379 consensus_queue.cc:1035] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:17.260428 11868 consensus_queue.cc:1035] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:17.260577 9441 consensus_peers.cc:485] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a -> Peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): Couldn't send request to peer 00a3f8e159c64327832d79fb32cdcca8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd6e2fa89d954521b69d097afcb4715b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:17.262324 10379 raft_consensus.cc:2932] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:17.262796 7020 raft_consensus.cc:2932] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:17.264705 9636 raft_consensus.cc:2932] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:17.264917 6745 catalog_manager.cc:5484] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:17.286581 7020 consensus_queue.cc:237] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:17.288061 6886 raft_consensus.cc:1261] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:17.288125 7278 raft_consensus.cc:1261] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:17.288504 9151 consensus_queue.cc:1035] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:17.288723 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d3eb284c0fc48b4912168c434844284. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:17.289005 9151 consensus_queue.cc:1035] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:17.291172 11596 raft_consensus.cc:2932] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:17.291622 7278 raft_consensus.cc:2932] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:17.293227 6886 raft_consensus.cc:2932] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:17.294179 6743 catalog_manager.cc:5484] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:17.308758 6745 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:17.308903 6745 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:17.311367 6745 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 22a5686856da4fc6b1d9707837a16922
I20240503 07:59:17.312011 9484 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a5686856da4fc6b1d9707837a16922 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:17 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:17.312050 9614 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a5686856da4fc6b1d9707837a16922 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:17 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:17.312361 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a5686856da4fc6b1d9707837a16922 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:17 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:17.312409 11933 tablet_replica.cc:330] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:17.312565 11933 raft_consensus.cc:2229] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:17.312712 11933 raft_consensus.cc:2258] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:17.312721 11934 tablet_replica.cc:330] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:17.312860 11934 raft_consensus.cc:2229] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:17.313208 11934 raft_consensus.cc:2258] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:17.313560 11936 tablet_replica.cc:330] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:17.313575 11933 ts_tablet_manager.cc:1875] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:17.314040 11934 ts_tablet_manager.cc:1875] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:17.315675 11936 raft_consensus.cc:2229] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:17.315892 11936 raft_consensus.cc:2258] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:17.316110 11933 ts_tablet_manager.cc:1888] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:17.316211 11933 log.cc:1196] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/22a5686856da4fc6b1d9707837a16922
I20240503 07:59:17.316519 11933 ts_tablet_manager.cc:1909] T 22a5686856da4fc6b1d9707837a16922 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:17.317059 11936 ts_tablet_manager.cc:1875] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:17.317271 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 22a5686856da4fc6b1d9707837a16922 (table rebalancer_dynamic_table_new_name [id=5bf71ba7bc3e45c48bbaee1c5f0f9098]) successfully deleted
I20240503 07:59:17.317286 9767 consensus_queue.cc:237] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:17.317942 11934 ts_tablet_manager.cc:1888] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:17.317997 11934 log.cc:1196] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/22a5686856da4fc6b1d9707837a16922
I20240503 07:59:17.318220 11934 ts_tablet_manager.cc:1909] T 22a5686856da4fc6b1d9707837a16922 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:17.318882 6886 raft_consensus.cc:1261] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.319085 9357 raft_consensus.cc:1261] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.319317 11879 consensus_queue.cc:1035] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:17.319824 9703 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): Couldn't send request to peer 00a3f8e159c64327832d79fb32cdcca8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e7d7f7ee4384a2ba8bd0dac0ddad757. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:17.319995 11879 consensus_queue.cc:1035] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:17.320694 11936 ts_tablet_manager.cc:1888] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:17.321424 11936 log.cc:1196] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/22a5686856da4fc6b1d9707837a16922
I20240503 07:59:17.321403 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 22a5686856da4fc6b1d9707837a16922 (table rebalancer_dynamic_table_new_name [id=5bf71ba7bc3e45c48bbaee1c5f0f9098]) successfully deleted
I20240503 07:59:17.321817 11936 ts_tablet_manager.cc:1909] T 22a5686856da4fc6b1d9707837a16922 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:17.321908 11343 raft_consensus.cc:2932] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:17.322383 9357 raft_consensus.cc:2932] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:17.323894 6886 raft_consensus.cc:2932] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:17.324857 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 22a5686856da4fc6b1d9707837a16922 (table rebalancer_dynamic_table_new_name [id=5bf71ba7bc3e45c48bbaee1c5f0f9098]) successfully deleted
I20240503 07:59:17.324826 6745 catalog_manager.cc:5484] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:17.347333 7020 consensus_queue.cc:237] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.348807 9636 raft_consensus.cc:1261] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.348877 7278 raft_consensus.cc:1261] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:17.348968 6956 consensus_peers.cc:485] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Couldn't send request to peer a372e98a670f49e9adca46ab5c836c4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a2312d0a7934056bf438a78e6641226. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:17.349226 11859 consensus_queue.cc:1035] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:17.349449 9151 consensus_queue.cc:1035] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:17.351933 9151 raft_consensus.cc:2932] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:17.353657 7278 raft_consensus.cc:2932] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:17.356510 6742 catalog_manager.cc:5484] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 2, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:17.358083 9636 raft_consensus.cc:2932] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:17.383924 11534 raft_consensus.cc:482] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.384063 11534 raft_consensus.cc:504] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:17.384560 11534 leader_election.cc:288] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:17.385412 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3503f0e1e29430a933986978978f1d4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:17.385574 9767 raft_consensus.cc:2454] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:17.385545 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3503f0e1e29430a933986978978f1d4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:17.385669 9636 raft_consensus.cc:2454] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:17.386101 6825 leader_election.cc:302] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:17.386279 11534 raft_consensus.cc:2781] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:17.386332 11534 raft_consensus.cc:482] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:17.386396 11534 raft_consensus.cc:3037] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.387521 11534 raft_consensus.cc:504] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:17.387900 11534 leader_election.cc:288] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:17.388290 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3503f0e1e29430a933986978978f1d4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:17.388351 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3503f0e1e29430a933986978978f1d4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:17.388427 9767 raft_consensus.cc:3037] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.388470 9636 raft_consensus.cc:3037] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.389451 9767 raft_consensus.cc:2454] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:17.389524 9636 raft_consensus.cc:2454] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:17.389834 6822 leader_election.cc:302] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:17.390002 11534 raft_consensus.cc:2781] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:17.390168 11534 raft_consensus.cc:686] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:17.390367 11534 consensus_queue.cc:237] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:17.393764 6745 catalog_manager.cc:5484] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:17.413962 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:17.420285 9484 tablet_service.cc:1467] Processing CreateTablet for tablet 6412ff543e85475eba85d9dfd060da51 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=94218707f6c14a4e99521367f3b8088e]), partition=
I20240503 07:59:17.420595 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6412ff543e85475eba85d9dfd060da51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.423187 11200 tablet_bootstrap.cc:492] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:17.424293 11200 tablet_bootstrap.cc:654] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.426872 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 6412ff543e85475eba85d9dfd060da51 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=94218707f6c14a4e99521367f3b8088e]), partition=
I20240503 07:59:17.427137 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6412ff543e85475eba85d9dfd060da51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.427186 11200 tablet_bootstrap.cc:492] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:17.427279 11200 ts_tablet_manager.cc:1367] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:17.427763 11200 raft_consensus.cc:348] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.427874 11200 raft_consensus.cc:374] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.427901 11200 raft_consensus.cc:729] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.428009 11200 consensus_queue.cc:260] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.428301 11200 ts_tablet_manager.cc:1398] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.429817 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 6412ff543e85475eba85d9dfd060da51 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=94218707f6c14a4e99521367f3b8088e]), partition=
I20240503 07:59:17.430058 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6412ff543e85475eba85d9dfd060da51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.432507 11832 tablet_bootstrap.cc:492] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:17.433415 11831 tablet_bootstrap.cc:492] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:17.433540 11832 tablet_bootstrap.cc:654] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.433957 11868 raft_consensus.cc:482] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.434052 11868 raft_consensus.cc:504] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.434456 11831 tablet_bootstrap.cc:654] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.434603 11868 leader_election.cc:288] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:17.435541 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6412ff543e85475eba85d9dfd060da51" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
W20240503 07:59:17.436347 9440 leader_election.cc:341] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:17.436753 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6412ff543e85475eba85d9dfd060da51" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
W20240503 07:59:17.438283 9441 leader_election.cc:341] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:17.438386 9441 leader_election.cc:302] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb73994515af4880b05383e78368bb4a; no voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:17.438710 11868 raft_consensus.cc:2726] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:17.439704 11831 tablet_bootstrap.cc:492] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:17.439816 11831 ts_tablet_manager.cc:1367] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 07:59:17.440404 11831 raft_consensus.cc:348] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.440513 11831 raft_consensus.cc:374] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.440542 11831 raft_consensus.cc:729] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.440687 11831 consensus_queue.cc:260] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.440986 11831 ts_tablet_manager.cc:1398] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:17.455324 11832 tablet_bootstrap.cc:492] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:17.455478 11832 ts_tablet_manager.cc:1367] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240503 07:59:17.456015 11832 raft_consensus.cc:348] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.456140 11832 raft_consensus.cc:374] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.456171 11832 raft_consensus.cc:729] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.456302 11832 consensus_queue.cc:260] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.456599 11832 ts_tablet_manager.cc:1398] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:17.471746 11880 raft_consensus.cc:482] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.471885 11880 raft_consensus.cc:504] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.472402 11880 leader_election.cc:288] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:17.472761 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23de5c1fe8c44b3c990f1128826a72b0" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:17.472841 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23de5c1fe8c44b3c990f1128826a72b0" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:17.472919 7020 raft_consensus.cc:2454] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:17.473011 9505 raft_consensus.cc:2454] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:17.473206 9702 leader_election.cc:302] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:17.473420 11880 raft_consensus.cc:2781] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:17.473481 11880 raft_consensus.cc:482] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:17.473506 11880 raft_consensus.cc:3037] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.474591 11880 raft_consensus.cc:504] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.475015 11880 leader_election.cc:288] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:17.475363 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23de5c1fe8c44b3c990f1128826a72b0" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:17.475478 9505 raft_consensus.cc:3037] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.475589 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23de5c1fe8c44b3c990f1128826a72b0" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:17.475699 7020 raft_consensus.cc:3037] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.476367 9505 raft_consensus.cc:2454] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:17.476612 7020 raft_consensus.cc:2454] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:17.476744 9703 leader_election.cc:302] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:17.476912 11880 raft_consensus.cc:2781] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:17.477017 11880 raft_consensus.cc:686] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:17.477159 11880 consensus_queue.cc:237] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.479662 6747 catalog_manager.cc:5484] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:17.496438 11918 raft_consensus.cc:482] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.496577 11918 raft_consensus.cc:504] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.497056 11918 leader_election.cc:288] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:17.497694 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6412ff543e85475eba85d9dfd060da51" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:17.497874 9767 raft_consensus.cc:2454] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:17.498101 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6412ff543e85475eba85d9dfd060da51" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:17.498245 9505 raft_consensus.cc:2454] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:17.499399 6956 leader_election.cc:302] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:17.499598 11918 raft_consensus.cc:2781] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:17.499663 11918 raft_consensus.cc:482] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:17.499696 11918 raft_consensus.cc:3037] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.501566 11918 raft_consensus.cc:504] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.501962 11918 leader_election.cc:288] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:17.503706 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:17.504879 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6412ff543e85475eba85d9dfd060da51" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:17.505031 9505 raft_consensus.cc:3037] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.506183 9505 raft_consensus.cc:2454] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:17.506502 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.507233 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6412ff543e85475eba85d9dfd060da51" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:17.507377 9767 raft_consensus.cc:3037] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.508513 9767 raft_consensus.cc:2454] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:17.510522 7129 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:17.511056 6956 leader_election.cc:302] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:17.511267 11918 raft_consensus.cc:2781] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:17.511355 11918 raft_consensus.cc:686] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:17.511562 11918 consensus_queue.cc:237] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.515144 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.525178 6747 catalog_manager.cc:5484] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:17.525671 9484 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:17.526582 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:17.529592 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:17.537741 9746 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:17.568522 11868 consensus_queue.cc:1035] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.573144 11868 consensus_queue.cc:1035] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.649701 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:17.654914 9767 raft_consensus.cc:1261] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.655865 9504 raft_consensus.cc:1261] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.656430 11918 consensus_queue.cc:1035] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.656749 11859 consensus_queue.cc:1035] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.659999 6746 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 6412ff543e85475eba85d9dfd060da51 (table rebalancer_dynamic_table [id=94218707f6c14a4e99521367f3b8088e]). Expected version 1 got 0
I20240503 07:59:17.660923 6743 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 6412ff543e85475eba85d9dfd060da51 (table rebalancer_dynamic_table [id=94218707f6c14a4e99521367f3b8088e]). Expected version 1 got 0
I20240503 07:59:17.664048 11789 tablet.cc:1722] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:17.664448 11789 tablet.cc:1620] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:17.665776 6746 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 6412ff543e85475eba85d9dfd060da51 (table rebalancer_dynamic_table [id=94218707f6c14a4e99521367f3b8088e]). Expected version 1 got 0
I20240503 07:59:17.667232 6743 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 6412ff543e85475eba85d9dfd060da51 (table rebalancer_dynamic_table [id=94218707f6c14a4e99521367f3b8088e]). Expected version 1 got 0
I20240503 07:59:17.671829 12070 tablet.cc:1722] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:17.672307 12070 tablet.cc:1620] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: MemRowSet was empty: no flush needed.
I20240503 07:59:17.674044 12071 tablet.cc:1722] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:17.674513 12071 tablet.cc:1620] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:17.674576 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.675030 6743 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=94218707f6c14a4e99521367f3b8088e] alter complete (version 1)
I20240503 07:59:17.676463 12074 ts_tablet_manager.cc:902] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8: Initiating tablet copy from peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:17.676800 7129 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:17.678445 12074 tablet_copy_client.cc:308] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:45435
I20240503 07:59:17.679461 9525 tablet_copy_service.cc:140] P cb73994515af4880b05383e78368bb4a: Received BeginTabletCopySession request for tablet bd6e2fa89d954521b69d097afcb4715b from peer 00a3f8e159c64327832d79fb32cdcca8 ({username='slave'} at 127.0.99.131:33047)
I20240503 07:59:17.679603 9525 tablet_copy_service.cc:161] P cb73994515af4880b05383e78368bb4a: Beginning new tablet copy session on tablet bd6e2fa89d954521b69d097afcb4715b from peer 00a3f8e159c64327832d79fb32cdcca8 at {username='slave'} at 127.0.99.131:33047: session id = 00a3f8e159c64327832d79fb32cdcca8-bd6e2fa89d954521b69d097afcb4715b
I20240503 07:59:17.679647 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.680999 9525 tablet_copy_source_session.cc:205] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:17.685134 12074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6e2fa89d954521b69d097afcb4715b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.688701 12074 tablet_copy_client.cc:783] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:17.688947 12074 tablet_copy_client.cc:647] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:17.693071 12074 tablet_copy_client.cc:523] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:17.695526 9746 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:17.695832 9484 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:17.696440 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:17.696684 12074 tablet_bootstrap.cc:492] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:17.702550 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:17.702589 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:17.707216 12074 tablet_bootstrap.cc:492] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.707646 12074 tablet_bootstrap.cc:492] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap complete.
I20240503 07:59:17.707909 12074 ts_tablet_manager.cc:1367] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240503 07:59:17.709743 12074 raft_consensus.cc:348] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:17.710009 12074 raft_consensus.cc:729] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.710232 12074 consensus_queue.cc:260] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:17.710733 12074 ts_tablet_manager.cc:1398] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:17.712069 9525 tablet_copy_service.cc:342] P cb73994515af4880b05383e78368bb4a: Request end of tablet copy session 00a3f8e159c64327832d79fb32cdcca8-bd6e2fa89d954521b69d097afcb4715b received from {username='slave'} at 127.0.99.131:33047
I20240503 07:59:17.713207 9525 tablet_copy_service.cc:433] P cb73994515af4880b05383e78368bb4a: ending tablet copy session 00a3f8e159c64327832d79fb32cdcca8-bd6e2fa89d954521b69d097afcb4715b on tablet bd6e2fa89d954521b69d097afcb4715b with peer 00a3f8e159c64327832d79fb32cdcca8
I20240503 07:59:17.748920 12081 ts_tablet_manager.cc:902] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:17.749600 12081 tablet_copy_client.cc:308] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46383
I20240503 07:59:17.755475 12088 ts_tablet_manager.cc:902] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92: Initiating tablet copy from peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:17.756847 12088 tablet_copy_client.cc:308] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:43683
I20240503 07:59:17.757081 12087 ts_tablet_manager.cc:902] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a: Initiating tablet copy from peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:17.757642 12087 tablet_copy_client.cc:308] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46383
I20240503 07:59:17.758193 7040 tablet_copy_service.cc:140] P 9c085a3241f14cb7ba1332b508f4b57d: Received BeginTabletCopySession request for tablet 7a2312d0a7934056bf438a78e6641226 from peer a372e98a670f49e9adca46ab5c836c4a ({username='slave'} at 127.0.99.136:40851)
I20240503 07:59:17.758299 7040 tablet_copy_service.cc:161] P 9c085a3241f14cb7ba1332b508f4b57d: Beginning new tablet copy session on tablet 7a2312d0a7934056bf438a78e6641226 from peer a372e98a670f49e9adca46ab5c836c4a at {username='slave'} at 127.0.99.136:40851: session id = a372e98a670f49e9adca46ab5c836c4a-7a2312d0a7934056bf438a78e6641226
I20240503 07:59:17.758530 9787 tablet_copy_service.cc:140] P a372e98a670f49e9adca46ab5c836c4a: Received BeginTabletCopySession request for tablet 40846ecb9f5d4c39a7bafac29c5d4e8a from peer c4218904dd7e42d2be95b901a6e5db92 ({username='slave'} at 127.0.99.129:37889)
I20240503 07:59:17.758661 9787 tablet_copy_service.cc:161] P a372e98a670f49e9adca46ab5c836c4a: Beginning new tablet copy session on tablet 40846ecb9f5d4c39a7bafac29c5d4e8a from peer c4218904dd7e42d2be95b901a6e5db92 at {username='slave'} at 127.0.99.129:37889: session id = c4218904dd7e42d2be95b901a6e5db92-40846ecb9f5d4c39a7bafac29c5d4e8a
I20240503 07:59:17.759313 7040 tablet_copy_source_session.cc:205] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:17.759645 7039 tablet_copy_service.cc:140] P 9c085a3241f14cb7ba1332b508f4b57d: Received BeginTabletCopySession request for tablet abb55c09be324f37a1a546e73008a80d from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:33039)
I20240503 07:59:17.759723 7039 tablet_copy_service.cc:161] P 9c085a3241f14cb7ba1332b508f4b57d: Beginning new tablet copy session on tablet abb55c09be324f37a1a546e73008a80d from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:33039: session id = 3befbb2d206f4be89d3e07f38837374b-abb55c09be324f37a1a546e73008a80d
I20240503 07:59:17.759806 9787 tablet_copy_source_session.cc:205] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:17.760639 7039 tablet_copy_source_session.cc:205] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:17.761214 12088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40846ecb9f5d4c39a7bafac29c5d4e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.761487 12081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abb55c09be324f37a1a546e73008a80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.762723 12087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2312d0a7934056bf438a78e6641226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.763950 12081 tablet_copy_client.cc:783] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:17.764106 12081 tablet_copy_client.cc:647] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:17.765714 12081 tablet_copy_client.cc:523] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:17.766849 12087 tablet_copy_client.cc:783] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:17.767053 12087 tablet_copy_client.cc:647] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:17.767100 12081 tablet_bootstrap.cc:492] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:17.766005 12088 tablet_copy_client.cc:783] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:17.767604 12088 tablet_copy_client.cc:647] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:17.768221 12087 tablet_copy_client.cc:523] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:17.769513 12088 tablet_copy_client.cc:523] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:17.769619 12087 tablet_bootstrap.cc:492] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:17.770473 12074 ts_tablet_manager.cc:902] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8: Initiating tablet copy from peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:17.770931 12074 tablet_copy_client.cc:308] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:43683
I20240503 07:59:17.771338 12088 tablet_bootstrap.cc:492] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:17.774181 9787 tablet_copy_service.cc:140] P a372e98a670f49e9adca46ab5c836c4a: Received BeginTabletCopySession request for tablet 3e7d7f7ee4384a2ba8bd0dac0ddad757 from peer 00a3f8e159c64327832d79fb32cdcca8 ({username='slave'} at 127.0.99.131:49907)
I20240503 07:59:17.774291 9787 tablet_copy_service.cc:161] P a372e98a670f49e9adca46ab5c836c4a: Beginning new tablet copy session on tablet 3e7d7f7ee4384a2ba8bd0dac0ddad757 from peer 00a3f8e159c64327832d79fb32cdcca8 at {username='slave'} at 127.0.99.131:49907: session id = 00a3f8e159c64327832d79fb32cdcca8-3e7d7f7ee4384a2ba8bd0dac0ddad757
I20240503 07:59:17.775317 9787 tablet_copy_source_session.cc:205] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:17.776810 12074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7d7f7ee4384a2ba8bd0dac0ddad757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.777489 12088 tablet_bootstrap.cc:492] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.777755 12088 tablet_bootstrap.cc:492] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92: Bootstrap complete.
I20240503 07:59:17.777927 12088 ts_tablet_manager.cc:1367] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 07:59:17.778424 12088 raft_consensus.cc:348] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:17.778553 12088 raft_consensus.cc:729] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.778690 12088 consensus_queue.cc:260] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:17.778975 12088 ts_tablet_manager.cc:1398] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.779501 12074 tablet_copy_client.cc:783] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:17.779625 12074 tablet_copy_client.cc:647] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:17.780040 9787 tablet_copy_service.cc:342] P a372e98a670f49e9adca46ab5c836c4a: Request end of tablet copy session c4218904dd7e42d2be95b901a6e5db92-40846ecb9f5d4c39a7bafac29c5d4e8a received from {username='slave'} at 127.0.99.129:37889
I20240503 07:59:17.780159 9787 tablet_copy_service.cc:433] P a372e98a670f49e9adca46ab5c836c4a: ending tablet copy session c4218904dd7e42d2be95b901a6e5db92-40846ecb9f5d4c39a7bafac29c5d4e8a on tablet 40846ecb9f5d4c39a7bafac29c5d4e8a with peer c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:17.781499 12074 tablet_copy_client.cc:523] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:17.782965 12074 tablet_bootstrap.cc:492] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:17.783305 12087 tablet_bootstrap.cc:492] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.783541 12087 tablet_bootstrap.cc:492] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap complete.
I20240503 07:59:17.783679 12087 ts_tablet_manager.cc:1367] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 07:59:17.784065 12087 raft_consensus.cc:348] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.784154 12087 raft_consensus.cc:729] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.784279 12087 consensus_queue.cc:260] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:17.785085 12087 ts_tablet_manager.cc:1398] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:17.785797 7039 tablet_copy_service.cc:342] P 9c085a3241f14cb7ba1332b508f4b57d: Request end of tablet copy session a372e98a670f49e9adca46ab5c836c4a-7a2312d0a7934056bf438a78e6641226 received from {username='slave'} at 127.0.99.136:40851
I20240503 07:59:17.785894 7039 tablet_copy_service.cc:433] P 9c085a3241f14cb7ba1332b508f4b57d: ending tablet copy session a372e98a670f49e9adca46ab5c836c4a-7a2312d0a7934056bf438a78e6641226 on tablet 7a2312d0a7934056bf438a78e6641226 with peer a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:17.787214 12074 tablet_bootstrap.cc:492] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.787453 12074 tablet_bootstrap.cc:492] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap complete.
I20240503 07:59:17.787616 12074 ts_tablet_manager.cc:1367] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 07:59:17.788129 12074 raft_consensus.cc:348] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:17.788247 12074 raft_consensus.cc:729] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.788376 12074 consensus_queue.cc:260] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:17.788645 12074 ts_tablet_manager.cc:1398] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.789252 9787 tablet_copy_service.cc:342] P a372e98a670f49e9adca46ab5c836c4a: Request end of tablet copy session 00a3f8e159c64327832d79fb32cdcca8-3e7d7f7ee4384a2ba8bd0dac0ddad757 received from {username='slave'} at 127.0.99.131:49907
I20240503 07:59:17.789201 6743 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:17.789331 9787 tablet_copy_service.cc:433] P a372e98a670f49e9adca46ab5c836c4a: ending tablet copy session 00a3f8e159c64327832d79fb32cdcca8-3e7d7f7ee4384a2ba8bd0dac0ddad757 on tablet 3e7d7f7ee4384a2ba8bd0dac0ddad757 with peer 00a3f8e159c64327832d79fb32cdcca8
I20240503 07:59:17.795153 12070 tablet.cc:1722] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:17.795516 12070 tablet.cc:1620] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: MemRowSet was empty: no flush needed.
I20240503 07:59:17.795918 11789 tablet.cc:1722] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:17.796240 11789 tablet.cc:1620] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:17.799142 12071 tablet.cc:1722] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:17.799479 12071 tablet.cc:1620] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:17.802084 12081 tablet_bootstrap.cc:492] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.804486 12081 tablet_bootstrap.cc:492] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:17.804769 12081 ts_tablet_manager.cc:1367] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.038s user 0.024s sys 0.002s
I20240503 07:59:17.805274 12081 raft_consensus.cc:348] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:17.805394 12081 raft_consensus.cc:729] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.805536 12081 consensus_queue.cc:260] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:17.805886 12081 ts_tablet_manager.cc:1398] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.805444 6743 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=94218707f6c14a4e99521367f3b8088e] alter complete (version 2)
I20240503 07:59:17.806943 7039 tablet_copy_service.cc:342] P 9c085a3241f14cb7ba1332b508f4b57d: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-abb55c09be324f37a1a546e73008a80d received from {username='slave'} at 127.0.99.133:33039
I20240503 07:59:17.807036 7039 tablet_copy_service.cc:433] P 9c085a3241f14cb7ba1332b508f4b57d: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-abb55c09be324f37a1a546e73008a80d on tablet abb55c09be324f37a1a546e73008a80d with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:17.877324 12081 ts_tablet_manager.cc:902] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:17.877856 12081 tablet_copy_client.cc:308] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46383
I20240503 07:59:17.878472 7039 tablet_copy_service.cc:140] P 9c085a3241f14cb7ba1332b508f4b57d: Received BeginTabletCopySession request for tablet 7d3eb284c0fc48b4912168c434844284 from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:33039)
I20240503 07:59:17.878598 7039 tablet_copy_service.cc:161] P 9c085a3241f14cb7ba1332b508f4b57d: Beginning new tablet copy session on tablet 7d3eb284c0fc48b4912168c434844284 from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:33039: session id = 3befbb2d206f4be89d3e07f38837374b-7d3eb284c0fc48b4912168c434844284
I20240503 07:59:17.881767 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.881992 7039 tablet_copy_source_session.cc:205] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:17.883209 12081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3eb284c0fc48b4912168c434844284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.884568 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.885787 12081 tablet_copy_client.cc:783] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:17.885941 12081 tablet_copy_client.cc:647] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:17.886209 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:17.894989 12081 tablet_copy_client.cc:523] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:17.895262 9484 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:17.901173 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:17.903458 7129 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:17.903568 11534 consensus_queue.cc:1035] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.904453 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:17.907644 6743 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:17.912016 12081 tablet_bootstrap.cc:492] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:17.918478 12070 tablet.cc:1722] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:17.919664 12071 tablet.cc:1722] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:17.920436 11789 tablet.cc:1722] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:17.926055 6743 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=94218707f6c14a4e99521367f3b8088e] alter complete (version 3)
I20240503 07:59:17.926497 9746 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:17.927645 11534 consensus_queue.cc:1035] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.949611 12081 tablet_bootstrap.cc:492] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.950024 12081 tablet_bootstrap.cc:492] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:17.950225 12081 ts_tablet_manager.cc:1367] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.038s user 0.026s sys 0.000s
I20240503 07:59:17.950784 12081 raft_consensus.cc:348] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:17.950935 12081 raft_consensus.cc:729] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.951125 12081 consensus_queue.cc:260] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:17.951997 12081 ts_tablet_manager.cc:1398] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:17.952447 7039 tablet_copy_service.cc:342] P 9c085a3241f14cb7ba1332b508f4b57d: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-7d3eb284c0fc48b4912168c434844284 received from {username='slave'} at 127.0.99.133:33039
I20240503 07:59:17.952527 7039 tablet_copy_service.cc:433] P 9c085a3241f14cb7ba1332b508f4b57d: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-7d3eb284c0fc48b4912168c434844284 on tablet 7d3eb284c0fc48b4912168c434844284 with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:17.979895 11880 consensus_queue.cc:1035] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.985070 11880 consensus_queue.cc:1035] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.032831 6747 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:18.032994 6747 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:18.036676 6747 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 6412ff543e85475eba85d9dfd060da51
I20240503 07:59:18.038578 9746 tablet_service.cc:1514] Processing DeleteTablet for tablet 6412ff543e85475eba85d9dfd060da51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:18 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:18.038753 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 6412ff543e85475eba85d9dfd060da51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:18 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:18.039520 9484 tablet_service.cc:1514] Processing DeleteTablet for tablet 6412ff543e85475eba85d9dfd060da51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:18 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:18.040477 12208 tablet_replica.cc:330] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:18.040638 12208 raft_consensus.cc:2229] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.040869 12209 tablet_replica.cc:330] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:18.041015 12209 raft_consensus.cc:2229] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:18.041273 12209 raft_consensus.cc:2258] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.040844 12208 raft_consensus.cc:2258] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.042217 12209 ts_tablet_manager.cc:1875] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:18.042433 12208 ts_tablet_manager.cc:1875] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:18.044888 12209 ts_tablet_manager.cc:1888] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:18.044940 12208 ts_tablet_manager.cc:1888] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:18.045051 12208 log.cc:1196] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/6412ff543e85475eba85d9dfd060da51
I20240503 07:59:18.045073 12209 log.cc:1196] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6412ff543e85475eba85d9dfd060da51
I20240503 07:59:18.045473 12209 ts_tablet_manager.cc:1909] T 6412ff543e85475eba85d9dfd060da51 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:18.046972 12208 ts_tablet_manager.cc:1909] T 6412ff543e85475eba85d9dfd060da51 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:18.047801 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 6412ff543e85475eba85d9dfd060da51 (table rebalancer_dynamic_table_new_name [id=94218707f6c14a4e99521367f3b8088e]) successfully deleted
I20240503 07:59:18.049985 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 6412ff543e85475eba85d9dfd060da51 (table rebalancer_dynamic_table_new_name [id=94218707f6c14a4e99521367f3b8088e]) successfully deleted
I20240503 07:59:18.050356 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:18.054405 12210 tablet_replica.cc:330] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:18.054642 12210 raft_consensus.cc:2229] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.054849 12210 raft_consensus.cc:2258] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.056457 12210 ts_tablet_manager.cc:1875] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:18.058236 9614 tablet_service.cc:1467] Processing CreateTablet for tablet a8c22545fc9f4b248c2fdc568967f47f (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a0ee201ab5cc47b798438b4276757b44]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.058501 9746 tablet_service.cc:1467] Processing CreateTablet for tablet a8c22545fc9f4b248c2fdc568967f47f (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a0ee201ab5cc47b798438b4276757b44]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.058615 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c22545fc9f4b248c2fdc568967f47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.058791 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c22545fc9f4b248c2fdc568967f47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.059473 12210 ts_tablet_manager.cc:1888] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:18.059543 12210 log.cc:1196] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/6412ff543e85475eba85d9dfd060da51
I20240503 07:59:18.059824 12210 ts_tablet_manager.cc:1909] T 6412ff543e85475eba85d9dfd060da51 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:18.061223 12211 tablet_bootstrap.cc:492] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:18.062067 9350 tablet_service.cc:1467] Processing CreateTablet for tablet fef7df80245c4b2c9fd3efd8331ba9e4 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a0ee201ab5cc47b798438b4276757b44]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.062407 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef7df80245c4b2c9fd3efd8331ba9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.062566 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 6412ff543e85475eba85d9dfd060da51 (table rebalancer_dynamic_table_new_name [id=94218707f6c14a4e99521367f3b8088e]) successfully deleted
I20240503 07:59:18.063195 9484 tablet_service.cc:1467] Processing CreateTablet for tablet a8c22545fc9f4b248c2fdc568967f47f (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a0ee201ab5cc47b798438b4276757b44]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.063237 7260 tablet_service.cc:1467] Processing CreateTablet for tablet fef7df80245c4b2c9fd3efd8331ba9e4 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a0ee201ab5cc47b798438b4276757b44]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.063278 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfe98aefcfe4380bb64f14df1a24318 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a0ee201ab5cc47b798438b4276757b44]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.063468 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c22545fc9f4b248c2fdc568967f47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.063469 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef7df80245c4b2c9fd3efd8331ba9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.064893 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfe98aefcfe4380bb64f14df1a24318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.062067 9353 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfe98aefcfe4380bb64f14df1a24318 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a0ee201ab5cc47b798438b4276757b44]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.065968 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfe98aefcfe4380bb64f14df1a24318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.066041 9615 tablet_service.cc:1467] Processing CreateTablet for tablet fef7df80245c4b2c9fd3efd8331ba9e4 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a0ee201ab5cc47b798438b4276757b44]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.066370 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef7df80245c4b2c9fd3efd8331ba9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.066810 9614 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfe98aefcfe4380bb64f14df1a24318 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a0ee201ab5cc47b798438b4276757b44]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.067143 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfe98aefcfe4380bb64f14df1a24318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.068846 12215 tablet_bootstrap.cc:492] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:18.069916 12212 tablet_bootstrap.cc:492] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:18.070034 12211 tablet_bootstrap.cc:654] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.070519 12215 tablet_bootstrap.cc:654] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.070849 12213 tablet_bootstrap.cc:492] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:18.070938 12212 tablet_bootstrap.cc:654] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.071316 12214 tablet_bootstrap.cc:492] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:18.072167 12212 tablet_bootstrap.cc:492] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:18.072273 12212 ts_tablet_manager.cc:1367] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:18.072307 12215 tablet_bootstrap.cc:492] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:18.072396 12215 ts_tablet_manager.cc:1367] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:18.072646 12214 tablet_bootstrap.cc:654] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.072726 12212 raft_consensus.cc:348] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:18.072880 12212 raft_consensus.cc:374] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.072917 12212 raft_consensus.cc:729] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.072882 12215 raft_consensus.cc:348] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:18.072993 12215 raft_consensus.cc:374] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.073025 12215 raft_consensus.cc:729] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.073045 12212 consensus_queue.cc:260] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:18.073161 12215 consensus_queue.cc:260] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:18.073354 12212 ts_tablet_manager.cc:1398] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.073428 12215 ts_tablet_manager.cc:1398] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.074043 12214 tablet_bootstrap.cc:492] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:18.074170 12214 ts_tablet_manager.cc:1367] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:18.074507 12213 tablet_bootstrap.cc:654] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.074623 12214 raft_consensus.cc:348] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:18.074712 12214 raft_consensus.cc:374] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.074730 12214 raft_consensus.cc:729] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.074860 12214 consensus_queue.cc:260] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:18.075145 12214 ts_tablet_manager.cc:1398] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.076789 12214 tablet_bootstrap.cc:492] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:18.077821 12214 tablet_bootstrap.cc:654] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.078701 12211 tablet_bootstrap.cc:492] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:18.079042 12211 ts_tablet_manager.cc:1367] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 07:59:18.079182 12214 tablet_bootstrap.cc:492] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:18.079334 12214 ts_tablet_manager.cc:1367] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:18.079618 12211 raft_consensus.cc:348] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:18.080216 12211 raft_consensus.cc:374] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.080318 12211 raft_consensus.cc:729] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.080580 12214 raft_consensus.cc:348] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.080554 12211 consensus_queue.cc:260] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:18.080760 12214 raft_consensus.cc:374] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.081337 12211 ts_tablet_manager.cc:1398] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:18.081604 12214 raft_consensus.cc:729] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.081533 12211 tablet_bootstrap.cc:492] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:18.081908 12214 consensus_queue.cc:260] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.082827 12211 tablet_bootstrap.cc:654] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.083585 12214 ts_tablet_manager.cc:1398] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:18.085354 12211 tablet_bootstrap.cc:492] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:18.085446 12211 ts_tablet_manager.cc:1367] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:18.085909 12211 raft_consensus.cc:348] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:18.086007 12211 raft_consensus.cc:374] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.086035 12211 raft_consensus.cc:729] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.086148 12211 consensus_queue.cc:260] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:18.087051 12213 tablet_bootstrap.cc:492] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:18.087095 12211 ts_tablet_manager.cc:1398] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:18.087172 12213 ts_tablet_manager.cc:1367] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20240503 07:59:18.087275 12211 tablet_bootstrap.cc:492] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:18.087657 12213 raft_consensus.cc:348] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:18.087832 12213 raft_consensus.cc:374] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.087881 12213 raft_consensus.cc:729] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.088025 12213 consensus_queue.cc:260] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:18.088327 12213 ts_tablet_manager.cc:1398] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:18.088461 12211 tablet_bootstrap.cc:654] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.089162 12213 tablet_bootstrap.cc:492] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:18.095009 9614 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:18.095479 11868 raft_consensus.cc:482] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.095597 11868 raft_consensus.cc:504] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:18.096028 12213 tablet_bootstrap.cc:654] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.096069 11868 leader_election.cc:288] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:18.096626 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c22545fc9f4b248c2fdc568967f47f" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:18.097132 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c22545fc9f4b248c2fdc568967f47f" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:18.096793 9766 raft_consensus.cc:2454] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:18.097271 9636 raft_consensus.cc:2454] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:18.097527 9441 leader_election.cc:302] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:18.097818 11868 raft_consensus.cc:2781] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.097879 11868 raft_consensus.cc:482] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.097914 11868 raft_consensus.cc:3037] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.099143 11868 raft_consensus.cc:504] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:18.099537 11868 leader_election.cc:288] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:18.099990 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c22545fc9f4b248c2fdc568967f47f" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:18.100123 9636 raft_consensus.cc:3037] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.101119 9636 raft_consensus.cc:2454] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:18.101197 12211 tablet_bootstrap.cc:492] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:18.101275 12211 ts_tablet_manager.cc:1367] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:18.101526 9438 leader_election.cc:302] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:18.101794 12211 raft_consensus.cc:348] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.101893 12211 raft_consensus.cc:374] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.101923 12211 raft_consensus.cc:729] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.102057 12211 consensus_queue.cc:260] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.102401 12211 ts_tablet_manager.cc:1398] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:18.102689 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c22545fc9f4b248c2fdc568967f47f" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:18.102818 9766 raft_consensus.cc:3037] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.106387 7129 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:18.108772 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:18.109282 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:18.111408 9766 raft_consensus.cc:2454] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:18.112793 11868 raft_consensus.cc:2781] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.112998 11868 raft_consensus.cc:686] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:18.113154 11868 consensus_queue.cc:237] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:18.115469 6744 catalog_manager.cc:5484] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:18.116515 12213 tablet_bootstrap.cc:492] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:18.116617 12213 ts_tablet_manager.cc:1367] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20240503 07:59:18.117105 12213 raft_consensus.cc:348] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.117245 12213 raft_consensus.cc:374] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.117285 12213 raft_consensus.cc:729] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.117403 12213 consensus_queue.cc:260] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.117657 12213 ts_tablet_manager.cc:1398] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:18.123482 9485 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:18.125263 9746 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:18.134829 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:18.143096 9357 raft_consensus.cc:1203] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:18.143093 6744 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:18.149122 9353 tablet_service.cc:1467] Processing CreateTablet for tablet cc5071c06b3b455897f364a8fd2b38c5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8fd33946a9dc4ee880d2c6dd3c943b32]), partition=
I20240503 07:59:18.149437 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5071c06b3b455897f364a8fd2b38c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.153842 9485 tablet_service.cc:1467] Processing CreateTablet for tablet cc5071c06b3b455897f364a8fd2b38c5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8fd33946a9dc4ee880d2c6dd3c943b32]), partition=
I20240503 07:59:18.154206 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5071c06b3b455897f364a8fd2b38c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.157696 12214 tablet_bootstrap.cc:492] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:18.158703 12214 tablet_bootstrap.cc:654] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.159632 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:18.162220 12213 tablet_bootstrap.cc:492] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:18.162220 6999 tablet_service.cc:1467] Processing CreateTablet for tablet cc5071c06b3b455897f364a8fd2b38c5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8fd33946a9dc4ee880d2c6dd3c943b32]), partition=
I20240503 07:59:18.162508 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5071c06b3b455897f364a8fd2b38c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.163455 12213 tablet_bootstrap.cc:654] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.163846 7149 raft_consensus.cc:1203] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:18.165107 12214 tablet_bootstrap.cc:492] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:18.165228 12214 ts_tablet_manager.cc:1367] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:18.165721 12214 raft_consensus.cc:348] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.165838 12214 raft_consensus.cc:374] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.165871 12214 raft_consensus.cc:729] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.165971 12213 tablet_bootstrap.cc:492] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:18.165979 12214 consensus_queue.cc:260] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.166069 12213 ts_tablet_manager.cc:1367] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:18.166267 12214 ts_tablet_manager.cc:1398] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.168324 12213 raft_consensus.cc:348] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.168448 12213 raft_consensus.cc:374] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.168476 12213 raft_consensus.cc:729] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.168634 12213 consensus_queue.cc:260] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.168905 12213 ts_tablet_manager.cc:1398] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:18.174628 12241 tablet_bootstrap.cc:492] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:18.182451 11512 raft_consensus.cc:482] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.182580 11512 raft_consensus.cc:504] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:18.183097 11512 leader_election.cc:288] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:18.183462 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef7df80245c4b2c9fd3efd8331ba9e4" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:18.183600 9357 raft_consensus.cc:2454] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:18.183967 9570 leader_election.cc:302] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:18.184176 11512 raft_consensus.cc:2781] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.184238 11512 raft_consensus.cc:482] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.184271 11512 raft_consensus.cc:3037] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.184414 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef7df80245c4b2c9fd3efd8331ba9e4" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:18.184545 7282 raft_consensus.cc:2454] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:18.185420 11512 raft_consensus.cc:504] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:18.185847 11512 leader_election.cc:288] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:18.186089 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef7df80245c4b2c9fd3efd8331ba9e4" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:18.186254 9357 raft_consensus.cc:3037] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.186270 12241 tablet_bootstrap.cc:654] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.186760 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef7df80245c4b2c9fd3efd8331ba9e4" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:18.186874 7282 raft_consensus.cc:3037] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.187346 9357 raft_consensus.cc:2454] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:18.187907 7282 raft_consensus.cc:2454] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:18.188401 9570 leader_election.cc:302] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:18.189174 6886 raft_consensus.cc:1203] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:18.190081 11512 raft_consensus.cc:2781] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.190285 11512 raft_consensus.cc:686] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:18.190462 11512 consensus_queue.cc:237] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:18.192385 6741 catalog_manager.cc:5484] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:18.199438 12241 tablet_bootstrap.cc:492] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:18.199877 12241 ts_tablet_manager.cc:1367] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 07:59:18.200498 12241 raft_consensus.cc:348] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.201010 12241 raft_consensus.cc:374] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.201118 12241 raft_consensus.cc:729] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.201332 12241 consensus_queue.cc:260] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.202070 12241 ts_tablet_manager.cc:1398] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:18.237463 11868 raft_consensus.cc:482] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.237622 11868 raft_consensus.cc:504] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.238098 11868 leader_election.cc:288] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:18.238777 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5071c06b3b455897f364a8fd2b38c5" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:18.238936 9357 raft_consensus.cc:2454] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:18.239396 9439 leader_election.cc:302] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:18.239518 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5071c06b3b455897f364a8fd2b38c5" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:18.239600 11868 raft_consensus.cc:2781] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.239652 11868 raft_consensus.cc:482] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.239655 7019 raft_consensus.cc:2454] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:18.239708 11868 raft_consensus.cc:3037] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.240887 11868 raft_consensus.cc:504] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.241290 11868 leader_election.cc:288] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:18.242413 11868 raft_consensus.cc:482] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.242496 11868 raft_consensus.cc:504] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.242501 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5071c06b3b455897f364a8fd2b38c5" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:18.242623 7019 raft_consensus.cc:3037] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.242635 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5071c06b3b455897f364a8fd2b38c5" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:18.242756 9357 raft_consensus.cc:3037] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.242844 11868 leader_election.cc:288] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:18.243696 7019 raft_consensus.cc:2454] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:18.243875 9357 raft_consensus.cc:2454] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:18.244040 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfe98aefcfe4380bb64f14df1a24318" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:18.244175 9636 raft_consensus.cc:2454] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:18.244547 9439 leader_election.cc:302] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:18.244557 9438 leader_election.cc:302] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:18.244750 11868 raft_consensus.cc:2781] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.244825 12235 raft_consensus.cc:2781] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.244874 12235 raft_consensus.cc:482] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.244904 11868 raft_consensus.cc:686] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:18.244907 12235 raft_consensus.cc:3037] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.245049 11868 consensus_queue.cc:237] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.245973 12235 raft_consensus.cc:504] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.246230 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfe98aefcfe4380bb64f14df1a24318" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:18.246310 12235 leader_election.cc:288] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:18.246384 9357 raft_consensus.cc:2454] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:18.246886 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfe98aefcfe4380bb64f14df1a24318" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:18.247020 9636 raft_consensus.cc:3037] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.247073 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfe98aefcfe4380bb64f14df1a24318" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:18.247174 9357 raft_consensus.cc:3037] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.247390 6744 catalog_manager.cc:5484] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:18.247798 9636 raft_consensus.cc:2454] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:18.248116 9357 raft_consensus.cc:2454] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:18.248420 9439 leader_election.cc:302] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:18.249104 12261 raft_consensus.cc:2781] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.249243 12261 raft_consensus.cc:686] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:18.249413 12261 consensus_queue.cc:237] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.251610 6744 catalog_manager.cc:5484] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:18.284435 7150 raft_consensus.cc:1203] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:18.306874 9357 raft_consensus.cc:1203] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:18.310734 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:18.311615 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:18.314985 9746 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:18.315819 7129 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:18.316962 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:18.317415 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:18.317562 9485 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:18.334738 9614 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:18.357319 9766 raft_consensus.cc:1203] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:18.363703 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:18.368465 7019 raft_consensus.cc:1261] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.368449 9357 raft_consensus.cc:1261] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.368894 12235 consensus_queue.cc:1035] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.369040 12261 consensus_queue.cc:1035] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.371987 11789 tablet.cc:1722] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:18.372077 6744 catalog_manager.cc:5576] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383) does not have the latest schema for tablet cc5071c06b3b455897f364a8fd2b38c5 (table rebalancer_dynamic_table [id=8fd33946a9dc4ee880d2c6dd3c943b32]). Expected version 1 got 0
I20240503 07:59:18.372336 11789 tablet.cc:1620] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:18.373019 12070 tablet.cc:1722] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:18.373347 12070 tablet.cc:1620] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: MemRowSet was empty: no flush needed.
I20240503 07:59:18.374596 6746 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet cc5071c06b3b455897f364a8fd2b38c5 (table rebalancer_dynamic_table [id=8fd33946a9dc4ee880d2c6dd3c943b32]). Expected version 1 got 0
I20240503 07:59:18.376143 6746 catalog_manager.cc:5576] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383) does not have the latest schema for tablet cc5071c06b3b455897f364a8fd2b38c5 (table rebalancer_dynamic_table [id=8fd33946a9dc4ee880d2c6dd3c943b32]). Expected version 1 got 0
I20240503 07:59:18.379783 6744 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=8fd33946a9dc4ee880d2c6dd3c943b32] alter complete (version 1)
I20240503 07:59:18.381335 12303 tablet.cc:1722] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:18.381671 12303 tablet.cc:1620] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:18.385555 6746 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet cc5071c06b3b455897f364a8fd2b38c5 (table rebalancer_dynamic_table [id=8fd33946a9dc4ee880d2c6dd3c943b32]). Expected version 1 got 0
I20240503 07:59:18.406499 7150 consensus_queue.cc:237] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:18.407917 9766 raft_consensus.cc:1261] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:18.407953 7019 raft_consensus.cc:1261] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:18.408315 12320 consensus_queue.cc:1035] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:18.408548 10716 consensus_queue.cc:1035] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:18.410595 10716 raft_consensus.cc:2932] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:18.411032 7019 raft_consensus.cc:2932] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:18.411321 9766 raft_consensus.cc:2932] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:18.414011 6746 catalog_manager.cc:5484] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:18.416832 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c06ab5a3e54ed2910afa50069810cc with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:18.417238 12324 tablet_replica.cc:330] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:18.417399 12324 raft_consensus.cc:2229] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.417572 12324 raft_consensus.cc:2258] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.418295 12324 ts_tablet_manager.cc:1875] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:18.421375 12324 ts_tablet_manager.cc:1888] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:18.421468 12324 log.cc:1196] T 16c06ab5a3e54ed2910afa50069810cc P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/16c06ab5a3e54ed2910afa50069810cc
I20240503 07:59:18.422071 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 16c06ab5a3e54ed2910afa50069810cc (table rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]) successfully deleted
I20240503 07:59:18.485283 6747 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:18.491811 11789 tablet.cc:1722] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:18.492182 11789 tablet.cc:1620] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:18.492359 12303 tablet.cc:1722] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:18.492455 12070 tablet.cc:1722] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:18.492658 12303 tablet.cc:1620] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:18.492736 12070 tablet.cc:1620] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: MemRowSet was empty: no flush needed.
I20240503 07:59:18.498612 6743 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=8fd33946a9dc4ee880d2c6dd3c943b32] alter complete (version 2)
I20240503 07:59:18.518005 7129 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:18.520416 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:18.523952 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:18.526367 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:18.527705 9746 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:18.529069 9614 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:18.532603 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:18.558441 9485 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:18.601387 6747 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:18.607921 11789 tablet.cc:1722] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:18.608947 12070 tablet.cc:1722] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:18.610852 12303 tablet.cc:1722] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:18.614337 6747 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=8fd33946a9dc4ee880d2c6dd3c943b32] alter complete (version 3)
I20240503 07:59:18.617945 12235 consensus_queue.cc:1035] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.618392 7150 consensus_queue.cc:237] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:18.619892 9357 raft_consensus.cc:1261] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:18.620054 7278 raft_consensus.cc:1261] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:18.620343 12320 consensus_queue.cc:1035] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:18.621378 12320 consensus_queue.cc:1035] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:18.621937 12261 consensus_queue.cc:1035] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.622953 12323 raft_consensus.cc:2932] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:18.625988 9357 raft_consensus.cc:2932] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:18.627847 7278 raft_consensus.cc:2932] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:18.629606 6738 catalog_manager.cc:5484] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index 4 to 5, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:18.633286 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet 5390e70b0449449aa3af05112bd29797 with delete_type TABLET_DATA_TOMBSTONED (TS c4218904dd7e42d2be95b901a6e5db92 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:18.642766 12405 tablet_replica.cc:330] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:18.643164 12405 raft_consensus.cc:2229] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.643375 12405 raft_consensus.cc:2258] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.644070 12405 ts_tablet_manager.cc:1875] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:18.646836 12405 ts_tablet_manager.cc:1888] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:18.646939 12405 log.cc:1196] T 5390e70b0449449aa3af05112bd29797 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5390e70b0449449aa3af05112bd29797
I20240503 07:59:18.647706 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 5390e70b0449449aa3af05112bd29797 (table rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]) successfully deleted
I20240503 07:59:18.683666 12243 consensus_queue.cc:1035] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.688198 11512 consensus_queue.cc:1035] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.719331 6745 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:18.719516 6745 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:18.722038 6745 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet cc5071c06b3b455897f364a8fd2b38c5
I20240503 07:59:18.722647 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5071c06b3b455897f364a8fd2b38c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:18 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:18.722783 9353 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5071c06b3b455897f364a8fd2b38c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:18 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:18.723006 12455 tablet_replica.cc:330] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:18.723050 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5071c06b3b455897f364a8fd2b38c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:18 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:18.723157 12455 raft_consensus.cc:2229] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.723274 12455 raft_consensus.cc:2258] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.723497 12457 tablet_replica.cc:330] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:18.723773 12457 raft_consensus.cc:2229] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:18.724052 12457 raft_consensus.cc:2258] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.724062 12455 ts_tablet_manager.cc:1875] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:18.724743 12457 ts_tablet_manager.cc:1875] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:18.727185 12455 ts_tablet_manager.cc:1888] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:18.727262 12455 log.cc:1196] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/cc5071c06b3b455897f364a8fd2b38c5
I20240503 07:59:18.727520 12455 ts_tablet_manager.cc:1909] T cc5071c06b3b455897f364a8fd2b38c5 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:18.728170 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet cc5071c06b3b455897f364a8fd2b38c5 (table rebalancer_dynamic_table_new_name [id=8fd33946a9dc4ee880d2c6dd3c943b32]) successfully deleted
I20240503 07:59:18.728263 12457 ts_tablet_manager.cc:1888] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:18.728377 12457 log.cc:1196] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/cc5071c06b3b455897f364a8fd2b38c5
I20240503 07:59:18.728617 12457 ts_tablet_manager.cc:1909] T cc5071c06b3b455897f364a8fd2b38c5 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:18.729318 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet cc5071c06b3b455897f364a8fd2b38c5 (table rebalancer_dynamic_table_new_name [id=8fd33946a9dc4ee880d2c6dd3c943b32]) successfully deleted
I20240503 07:59:18.730943 12456 tablet_replica.cc:330] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:18.731119 12456 raft_consensus.cc:2229] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.732812 12456 raft_consensus.cc:2258] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.733628 12456 ts_tablet_manager.cc:1875] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:18.741814 7129 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:18.743480 12456 ts_tablet_manager.cc:1888] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:18.743695 12456 log.cc:1196] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/cc5071c06b3b455897f364a8fd2b38c5
I20240503 07:59:18.745314 12456 ts_tablet_manager.cc:1909] T cc5071c06b3b455897f364a8fd2b38c5 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:18.747009 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet cc5071c06b3b455897f364a8fd2b38c5 (table rebalancer_dynamic_table_new_name [id=8fd33946a9dc4ee880d2c6dd3c943b32]) successfully deleted
I20240503 07:59:18.752439 7260 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:18.753427 9614 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:18.756788 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:18.757285 9746 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:18.762713 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:18.772011 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:18.779780 9485 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:18.790911 6745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:18.798919 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 27139acdbdb54c248f35fcf9ec023f4d (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=fb966e884afa4325a2dc1798eb47fbf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.798988 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 350265c4f7af409481c60802a6122f7f (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=fb966e884afa4325a2dc1798eb47fbf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.799288 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350265c4f7af409481c60802a6122f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.799289 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27139acdbdb54c248f35fcf9ec023f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.801976 9614 tablet_service.cc:1467] Processing CreateTablet for tablet 27139acdbdb54c248f35fcf9ec023f4d (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=fb966e884afa4325a2dc1798eb47fbf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.802289 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27139acdbdb54c248f35fcf9ec023f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.804675 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 136fc500a47a4d9f98e48c5a0c7af17e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=fb966e884afa4325a2dc1798eb47fbf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.804934 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136fc500a47a4d9f98e48c5a0c7af17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.806727 12261 consensus_queue.cc:1035] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.807217 9353 tablet_service.cc:1467] Processing CreateTablet for tablet 27139acdbdb54c248f35fcf9ec023f4d (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=fb966e884afa4325a2dc1798eb47fbf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.807561 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27139acdbdb54c248f35fcf9ec023f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.809976 12472 tablet_bootstrap.cc:492] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:18.811092 12472 tablet_bootstrap.cc:654] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.812594 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 136fc500a47a4d9f98e48c5a0c7af17e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=fb966e884afa4325a2dc1798eb47fbf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.812912 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136fc500a47a4d9f98e48c5a0c7af17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.813277 12472 tablet_bootstrap.cc:492] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:18.813359 12472 ts_tablet_manager.cc:1367] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:18.813804 12472 raft_consensus.cc:348] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:18.813915 12472 raft_consensus.cc:374] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.813942 12472 raft_consensus.cc:729] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.814059 12472 consensus_queue.cc:260] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:18.813056 12470 tablet_bootstrap.cc:492] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:18.815297 12470 tablet_bootstrap.cc:654] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.815366 12472 ts_tablet_manager.cc:1398] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:18.816046 12469 tablet_bootstrap.cc:492] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:18.816318 9484 tablet_service.cc:1467] Processing CreateTablet for tablet 350265c4f7af409481c60802a6122f7f (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=fb966e884afa4325a2dc1798eb47fbf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.816592 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350265c4f7af409481c60802a6122f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.817559 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 350265c4f7af409481c60802a6122f7f (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=fb966e884afa4325a2dc1798eb47fbf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.817849 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350265c4f7af409481c60802a6122f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.819934 12476 tablet_bootstrap.cc:492] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:18.821023 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 136fc500a47a4d9f98e48c5a0c7af17e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=fb966e884afa4325a2dc1798eb47fbf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.821322 12476 tablet_bootstrap.cc:654] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.821542 12469 tablet_bootstrap.cc:654] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.824175 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:18.824348 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136fc500a47a4d9f98e48c5a0c7af17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.825755 12469 tablet_bootstrap.cc:492] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:18.822770 12470 tablet_bootstrap.cc:492] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:18.825951 12470 ts_tablet_manager.cc:1367] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:18.826571 12474 tablet_bootstrap.cc:492] T 136fc500a47a4d9f98e48c5a0c7af17e P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:18.826608 12470 raft_consensus.cc:348] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:18.826882 12469 ts_tablet_manager.cc:1367] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:18.827430 12469 raft_consensus.cc:348] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.827798 12474 tablet_bootstrap.cc:654] T 136fc500a47a4d9f98e48c5a0c7af17e P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.828089 12469 raft_consensus.cc:374] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.828213 12470 raft_consensus.cc:374] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.829005 12476 tablet_bootstrap.cc:492] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:18.829109 12476 ts_tablet_manager.cc:1367] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:18.829329 12474 tablet_bootstrap.cc:492] T 136fc500a47a4d9f98e48c5a0c7af17e P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:18.829418 12474 ts_tablet_manager.cc:1367] T 136fc500a47a4d9f98e48c5a0c7af17e P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:18.829629 12476 raft_consensus.cc:348] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.829748 12476 raft_consensus.cc:374] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.829777 12476 raft_consensus.cc:729] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.829906 12476 consensus_queue.cc:260] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.829924 12474 raft_consensus.cc:348] T 136fc500a47a4d9f98e48c5a0c7af17e P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.830035 12474 raft_consensus.cc:374] T 136fc500a47a4d9f98e48c5a0c7af17e P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.830065 12474 raft_consensus.cc:729] T 136fc500a47a4d9f98e48c5a0c7af17e P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.830206 12476 ts_tablet_manager.cc:1398] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.830183 12474 consensus_queue.cc:260] T 136fc500a47a4d9f98e48c5a0c7af17e P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.831070 9746 tablet_service.cc:1467] Processing CreateTablet for tablet dd7254c992bf4cbeb363b763d02e60a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=265b99f91f04467abc77330f48071a62]), partition=
I20240503 07:59:18.831416 9484 tablet_service.cc:1467] Processing CreateTablet for tablet dd7254c992bf4cbeb363b763d02e60a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=265b99f91f04467abc77330f48071a62]), partition=
I20240503 07:59:18.832899 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7254c992bf4cbeb363b763d02e60a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.832949 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7254c992bf4cbeb363b763d02e60a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.834152 12474 ts_tablet_manager.cc:1398] T 136fc500a47a4d9f98e48c5a0c7af17e P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:18.832072 11868 consensus_queue.cc:1035] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.836565 12474 tablet_bootstrap.cc:492] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:18.832774 9615 tablet_service.cc:1467] Processing CreateTablet for tablet dd7254c992bf4cbeb363b763d02e60a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=265b99f91f04467abc77330f48071a62]), partition=
I20240503 07:59:18.837374 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7254c992bf4cbeb363b763d02e60a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.839008 12474 tablet_bootstrap.cc:654] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.832343 12469 raft_consensus.cc:729] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.840540 12470 raft_consensus.cc:729] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.840729 12470 consensus_queue.cc:260] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:18.841082 12470 ts_tablet_manager.cc:1398] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240503 07:59:18.841364 12474 tablet_bootstrap.cc:492] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:18.841300 12469 consensus_queue.cc:260] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.841503 12474 ts_tablet_manager.cc:1367] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:18.842085 12469 ts_tablet_manager.cc:1398] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240503 07:59:18.842074 12474 raft_consensus.cc:348] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.842299 12474 raft_consensus.cc:374] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.842375 12469 tablet_bootstrap.cc:492] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:18.842396 12474 raft_consensus.cc:729] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.842594 12474 consensus_queue.cc:260] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.843009 12474 ts_tablet_manager.cc:1398] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:18.843331 12469 tablet_bootstrap.cc:654] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.845605 12470 tablet_bootstrap.cc:492] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:18.846747 12470 tablet_bootstrap.cc:654] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.848340 12469 tablet_bootstrap.cc:492] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:18.848431 12469 ts_tablet_manager.cc:1367] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:18.848909 12469 raft_consensus.cc:348] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:18.849013 12469 raft_consensus.cc:374] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.849033 12469 raft_consensus.cc:729] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.849131 12469 consensus_queue.cc:260] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:18.849413 12469 ts_tablet_manager.cc:1398] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.849574 12469 tablet_bootstrap.cc:492] T 136fc500a47a4d9f98e48c5a0c7af17e P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:18.852216 12469 tablet_bootstrap.cc:654] T 136fc500a47a4d9f98e48c5a0c7af17e P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.852583 11512 raft_consensus.cc:482] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.852660 11512 raft_consensus.cc:504] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:18.853055 11512 leader_election.cc:288] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:18.853436 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27139acdbdb54c248f35fcf9ec023f4d" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:18.853570 9504 raft_consensus.cc:2454] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:18.853636 12469 tablet_bootstrap.cc:492] T 136fc500a47a4d9f98e48c5a0c7af17e P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:18.853729 12469 ts_tablet_manager.cc:1367] T 136fc500a47a4d9f98e48c5a0c7af17e P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:18.853904 9572 leader_election.cc:302] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:18.854171 11512 raft_consensus.cc:2781] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.854231 11512 raft_consensus.cc:482] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.854269 11512 raft_consensus.cc:3037] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.854653 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27139acdbdb54c248f35fcf9ec023f4d" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:18.854597 12469 raft_consensus.cc:348] T 136fc500a47a4d9f98e48c5a0c7af17e P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.854801 9357 raft_consensus.cc:2454] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:18.854815 12469 raft_consensus.cc:374] T 136fc500a47a4d9f98e48c5a0c7af17e P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.854856 12469 raft_consensus.cc:729] T 136fc500a47a4d9f98e48c5a0c7af17e P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.854981 12469 consensus_queue.cc:260] T 136fc500a47a4d9f98e48c5a0c7af17e P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.855217 12470 tablet_bootstrap.cc:492] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:18.855319 12470 ts_tablet_manager.cc:1367] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:18.855403 11512 raft_consensus.cc:504] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:18.855728 12469 ts_tablet_manager.cc:1398] T 136fc500a47a4d9f98e48c5a0c7af17e P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:18.855814 12470 raft_consensus.cc:348] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.855904 12469 tablet_bootstrap.cc:492] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:18.855967 12470 raft_consensus.cc:374] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.856021 12470 raft_consensus.cc:729] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.856169 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27139acdbdb54c248f35fcf9ec023f4d" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:18.856267 9357 raft_consensus.cc:3037] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.856863 12469 tablet_bootstrap.cc:654] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.857024 9357 raft_consensus.cc:2454] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:18.856112 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27139acdbdb54c248f35fcf9ec023f4d" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:18.857295 9504 raft_consensus.cc:3037] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.855741 11512 leader_election.cc:288] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:18.857949 9570 leader_election.cc:302] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:18.858096 11512 raft_consensus.cc:2781] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.858172 11512 raft_consensus.cc:686] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:18.858381 9504 raft_consensus.cc:2454] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:18.858295 11512 consensus_queue.cc:237] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:18.859730 12469 tablet_bootstrap.cc:492] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:18.859805 12469 ts_tablet_manager.cc:1367] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:18.860152 12470 consensus_queue.cc:260] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.860224 12469 raft_consensus.cc:348] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.860363 12469 raft_consensus.cc:374] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.860446 12469 raft_consensus.cc:729] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.860607 12469 consensus_queue.cc:260] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.860960 12469 ts_tablet_manager.cc:1398] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.861366 6744 catalog_manager.cc:5484] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:18.861754 12470 ts_tablet_manager.cc:1398] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:18.864866 12470 tablet_bootstrap.cc:492] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:18.865926 12470 tablet_bootstrap.cc:654] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.866984 12470 tablet_bootstrap.cc:492] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:18.867111 12470 ts_tablet_manager.cc:1367] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:18.867520 12470 raft_consensus.cc:348] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.867699 12470 raft_consensus.cc:374] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.867806 12470 raft_consensus.cc:729] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.868013 12470 consensus_queue.cc:260] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:18.868361 12470 ts_tablet_manager.cc:1398] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.868572 12470 tablet_bootstrap.cc:492] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:18.870079 12470 tablet_bootstrap.cc:654] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.871564 12470 tablet_bootstrap.cc:492] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:18.871651 12470 ts_tablet_manager.cc:1367] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:18.872082 12470 raft_consensus.cc:348] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.872184 12470 raft_consensus.cc:374] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.872215 12470 raft_consensus.cc:729] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.872328 12470 consensus_queue.cc:260] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.872565 12470 ts_tablet_manager.cc:1398] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.879971 9766 consensus_queue.cc:237] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:18.881357 6886 raft_consensus.cc:1261] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:18.881397 9635 raft_consensus.cc:1261] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:18.881807 12176 consensus_queue.cc:1035] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:18.882059 12377 consensus_queue.cc:1035] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:18.883733 11881 raft_consensus.cc:482] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.883863 11881 raft_consensus.cc:504] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.884281 11881 leader_election.cc:288] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:18.884516 12176 raft_consensus.cc:2932] T 40846ecb9f5d4c39a7bafac29c5d4e8a P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:18.884709 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350265c4f7af409481c60802a6122f7f" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:18.884860 9504 raft_consensus.cc:2454] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:18.885027 6888 raft_consensus.cc:2932] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:18.885164 7218 leader_election.cc:302] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:18.885336 11881 raft_consensus.cc:2781] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.885390 11881 raft_consensus.cc:482] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.885432 11881 raft_consensus.cc:3037] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.886473 11881 raft_consensus.cc:504] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.886844 11881 leader_election.cc:288] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:18.887324 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350265c4f7af409481c60802a6122f7f" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:18.887444 9636 raft_consensus.cc:3037] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.887492 9634 raft_consensus.cc:2932] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:18.888093 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350265c4f7af409481c60802a6122f7f" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:18.888469 9636 raft_consensus.cc:2454] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:18.889886 6744 catalog_manager.cc:5484] T 40846ecb9f5d4c39a7bafac29c5d4e8a P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:18.891760 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350265c4f7af409481c60802a6122f7f" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:18.891891 9504 raft_consensus.cc:3037] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.892748 9504 raft_consensus.cc:2454] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:18.893004 9353 tablet_service.cc:1514] Processing DeleteTablet for tablet 40846ecb9f5d4c39a7bafac29c5d4e8a with delete_type TABLET_DATA_TOMBSTONED (TS 3befbb2d206f4be89d3e07f38837374b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:18.893074 7218 leader_election.cc:302] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:18.893285 12456 tablet_replica.cc:330] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:18.893277 11881 raft_consensus.cc:2781] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.893357 11881 raft_consensus.cc:686] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:18.893426 12456 raft_consensus.cc:2229] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.893486 11881 consensus_queue.cc:237] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:18.893548 12456 raft_consensus.cc:2258] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.894037 12456 ts_tablet_manager.cc:1875] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:18.896508 12456 ts_tablet_manager.cc:1888] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:18.896587 12456 log.cc:1196] T 40846ecb9f5d4c39a7bafac29c5d4e8a P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/40846ecb9f5d4c39a7bafac29c5d4e8a
I20240503 07:59:18.896512 6738 catalog_manager.cc:5484] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:18.897188 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 40846ecb9f5d4c39a7bafac29c5d4e8a (table rebalancer_extra_table_12 [id=4b3fe402f6344742a8841ffd09a9f966]) successfully deleted
I20240503 07:59:19.099975 11868 raft_consensus.cc:482] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.100113 11868 raft_consensus.cc:504] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.100600 11868 leader_election.cc:288] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:19.100926 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136fc500a47a4d9f98e48c5a0c7af17e" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:19.101011 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136fc500a47a4d9f98e48c5a0c7af17e" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:19.101083 9634 raft_consensus.cc:2454] T 136fc500a47a4d9f98e48c5a0c7af17e P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:19.101135 9766 raft_consensus.cc:2454] T 136fc500a47a4d9f98e48c5a0c7af17e P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:19.101482 9438 leader_election.cc:302] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:19.101667 11868 raft_consensus.cc:2781] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.101718 11868 raft_consensus.cc:482] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.101764 11868 raft_consensus.cc:3037] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.102982 11868 raft_consensus.cc:504] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.103377 11868 leader_election.cc:288] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:19.103844 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136fc500a47a4d9f98e48c5a0c7af17e" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:19.103837 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136fc500a47a4d9f98e48c5a0c7af17e" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:19.103965 9766 raft_consensus.cc:3037] T 136fc500a47a4d9f98e48c5a0c7af17e P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.103964 9634 raft_consensus.cc:3037] T 136fc500a47a4d9f98e48c5a0c7af17e P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.104810 9766 raft_consensus.cc:2454] T 136fc500a47a4d9f98e48c5a0c7af17e P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:19.104846 9634 raft_consensus.cc:2454] T 136fc500a47a4d9f98e48c5a0c7af17e P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:19.105084 9441 leader_election.cc:302] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:19.105249 11868 raft_consensus.cc:2781] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.105331 11868 raft_consensus.cc:686] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:19.105465 11868 consensus_queue.cc:237] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.107717 6742 catalog_manager.cc:5484] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.119947 11868 raft_consensus.cc:482] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.120080 11868 raft_consensus.cc:504] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.120501 11868 leader_election.cc:288] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:19.120944 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7254c992bf4cbeb363b763d02e60a1" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:19.120944 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7254c992bf4cbeb363b763d02e60a1" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:19.121102 9766 raft_consensus.cc:2454] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:19.121114 9636 raft_consensus.cc:2454] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:19.121393 9441 leader_election.cc:302] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:19.121599 11868 raft_consensus.cc:2781] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.121656 11868 raft_consensus.cc:482] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.121685 11868 raft_consensus.cc:3037] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.122573 11868 raft_consensus.cc:504] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.122916 11868 leader_election.cc:288] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:19.123272 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7254c992bf4cbeb363b763d02e60a1" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:19.123318 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7254c992bf4cbeb363b763d02e60a1" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:19.123411 9766 raft_consensus.cc:3037] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.123457 9636 raft_consensus.cc:3037] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.124289 9766 raft_consensus.cc:2454] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:19.124322 9636 raft_consensus.cc:2454] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:19.124630 9438 leader_election.cc:302] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:19.124801 11868 raft_consensus.cc:2781] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.124886 11868 raft_consensus.cc:686] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:19.125021 11868 consensus_queue.cc:237] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.126900 6742 catalog_manager.cc:5484] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:19.160434 9353 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:19.161552 7129 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:19.163656 9615 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:19.170117 9746 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:19.170771 9485 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:19.187521 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:19.190088 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:19.205247 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:19.238860 6742 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:19.244320 9636 raft_consensus.cc:1261] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.244320 9766 raft_consensus.cc:1261] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.244894 12235 consensus_queue.cc:1035] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.245329 11868 consensus_queue.cc:1035] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.249181 6746 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet dd7254c992bf4cbeb363b763d02e60a1 (table rebalancer_dynamic_table [id=265b99f91f04467abc77330f48071a62]). Expected version 1 got 0
I20240503 07:59:19.250151 6742 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet dd7254c992bf4cbeb363b763d02e60a1 (table rebalancer_dynamic_table [id=265b99f91f04467abc77330f48071a62]). Expected version 1 got 0
I20240503 07:59:19.250968 12578 tablet.cc:1722] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:19.251441 12578 tablet.cc:1620] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:19.252137 6742 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet dd7254c992bf4cbeb363b763d02e60a1 (table rebalancer_dynamic_table [id=265b99f91f04467abc77330f48071a62]). Expected version 1 got 0
I20240503 07:59:19.252502 6746 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet dd7254c992bf4cbeb363b763d02e60a1 (table rebalancer_dynamic_table [id=265b99f91f04467abc77330f48071a62]). Expected version 1 got 0
I20240503 07:59:19.256090 12579 tablet.cc:1722] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:19.256467 12579 tablet.cc:1620] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:19.256871 12580 tablet.cc:1722] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:19.257215 12580 tablet.cc:1620] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:19.257750 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=265b99f91f04467abc77330f48071a62] alter complete (version 1)
I20240503 07:59:19.279477 9504 consensus_queue.cc:237] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:19.281024 7020 raft_consensus.cc:1261] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:19.281412 7150 raft_consensus.cc:1261] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:19.281414 11868 consensus_queue.cc:1035] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:19.282058 11868 consensus_queue.cc:1035] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:19.283321 12235 raft_consensus.cc:2932] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:19.283790 7020 raft_consensus.cc:2932] T bd6e2fa89d954521b69d097afcb4715b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:19.284175 7150 raft_consensus.cc:2932] T bd6e2fa89d954521b69d097afcb4715b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:19.286568 11881 consensus_queue.cc:1035] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.287075 6742 catalog_manager.cc:5484] T bd6e2fa89d954521b69d097afcb4715b P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:19.290572 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6e2fa89d954521b69d097afcb4715b with delete_type TABLET_DATA_TOMBSTONED (TS 159177b3caeb434baa98e04136d050b2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:19.291657 12238 consensus_queue.cc:1035] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.291898 12596 tablet_replica.cc:330] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:19.292034 12596 raft_consensus.cc:2229] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:19.292177 12596 raft_consensus.cc:2258] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:19.292634 12596 ts_tablet_manager.cc:1875] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:19.295841 12596 ts_tablet_manager.cc:1888] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:19.295917 12596 log.cc:1196] T bd6e2fa89d954521b69d097afcb4715b P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/bd6e2fa89d954521b69d097afcb4715b
I20240503 07:59:19.296424 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet bd6e2fa89d954521b69d097afcb4715b (table rebalancer_extra_table_6 [id=476bb7319d81414d9f735392a591b6a6]) successfully deleted
I20240503 07:59:19.352246 7150 consensus_queue.cc:237] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
W20240503 07:59:19.354108 7086 consensus_peers.cc:485] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): Couldn't send request to peer 9c085a3241f14cb7ba1332b508f4b57d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a838dc6e3114be5aceac51d567bc645. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.354108 9636 raft_consensus.cc:1261] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.354552 9504 raft_consensus.cc:1261] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.354597 12644 consensus_queue.cc:1035] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.354979 12644 consensus_queue.cc:1035] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.356784 12645 raft_consensus.cc:2932] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:19.357208 9504 raft_consensus.cc:2932] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:19.358889 9636 raft_consensus.cc:2932] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:19.361109 6742 catalog_manager.cc:5484] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:19.361796 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:19.368245 12578 tablet.cc:1722] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:19.368541 12580 tablet.cc:1722] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:19.368602 12578 tablet.cc:1620] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:19.368851 12580 tablet.cc:1620] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:19.369580 12579 tablet.cc:1722] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:19.369868 12579 tablet.cc:1620] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:19.374953 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=265b99f91f04467abc77330f48071a62] alter complete (version 2)
I20240503 07:59:19.394932 11512 consensus_queue.cc:1035] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.395279 7150 consensus_queue.cc:237] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:19.397279 7278 raft_consensus.cc:1261] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:19.397385 7020 raft_consensus.cc:1261] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:19.397986 12644 consensus_queue.cc:1035] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:19.398391 12644 consensus_queue.cc:1035] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:19.398650 7087 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Couldn't send request to peer a372e98a670f49e9adca46ab5c836c4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c2386cd009847d9a1fe7a959f47086c. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.402508 12644 raft_consensus.cc:2932] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:19.403091 12671 consensus_queue.cc:1035] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.403012 7020 raft_consensus.cc:2932] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:19.405179 7278 raft_consensus.cc:2932] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:19.409840 6738 catalog_manager.cc:5484] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:19.434252 7150 consensus_queue.cc:237] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.436236 7020 raft_consensus.cc:1261] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:19.436390 6888 raft_consensus.cc:1261] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:19.436880 12320 consensus_queue.cc:1035] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:19.437152 12645 consensus_queue.cc:1035] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:19.437371 7087 consensus_peers.cc:485] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c27273ae45c449018143cc55e65336a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.439785 12645 raft_consensus.cc:2932] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.440322 6888 raft_consensus.cc:2932] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.444293 6739 catalog_manager.cc:5484] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.444949 7020 raft_consensus.cc:2932] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.470319 7020 consensus_queue.cc:237] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.471864 7150 raft_consensus.cc:1261] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:19.471885 7278 raft_consensus.cc:1261] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:19.471992 6953 consensus_peers.cc:485] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea625a1834fb4a038a32a90d2fcbf1ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.472295 12707 consensus_queue.cc:1035] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:19.472510 12569 consensus_queue.cc:1035] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:19.474399 12569 raft_consensus.cc:2932] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:19.474882 7150 raft_consensus.cc:2932] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:19.476452 7278 raft_consensus.cc:2932] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:19.478583 6739 catalog_manager.cc:5484] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:19.480266 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:19.486697 12578 tablet.cc:1722] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:19.487385 12580 tablet.cc:1722] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:19.488533 12579 tablet.cc:1722] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:19.493088 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=265b99f91f04467abc77330f48071a62] alter complete (version 3)
I20240503 07:59:19.507673 6888 consensus_queue.cc:237] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:19.509263 7278 raft_consensus.cc:1261] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:19.509322 7150 raft_consensus.cc:1261] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:19.509346 6825 consensus_peers.cc:485] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 -> Peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Couldn't send request to peer a372e98a670f49e9adca46ab5c836c4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b794508aa77492398e48d25edfb8aa3. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.509683 12726 consensus_queue.cc:1035] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:19.509881 11974 consensus_queue.cc:1035] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:19.511698 11974 raft_consensus.cc:2932] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:19.512648 7278 raft_consensus.cc:2932] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:19.515288 7150 raft_consensus.cc:2932] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:19.515766 6740 catalog_manager.cc:5484] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:19.536139 7278 consensus_queue.cc:237] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.537860 7020 raft_consensus.cc:1261] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:19.537873 6888 raft_consensus.cc:1261] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:19.538138 7218 consensus_peers.cc:485] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa45918bb38c4583ad59524bda2d48ee. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.538311 12746 consensus_queue.cc:1035] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:19.538551 11881 consensus_queue.cc:1035] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:19.540522 12746 raft_consensus.cc:2932] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.540961 7020 raft_consensus.cc:2932] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.543367 6888 raft_consensus.cc:2932] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.544030 6738 catalog_manager.cc:5484] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.564755 7150 consensus_queue.cc:237] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:19.566167 6888 raft_consensus.cc:1261] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:19.566200 7278 raft_consensus.cc:1261] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:19.566289 7085 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 795f69249afe444ba0828875e1839b8c. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.566696 12320 consensus_queue.cc:1035] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:19.566876 12644 consensus_queue.cc:1035] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:19.568708 12644 raft_consensus.cc:2932] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:19.569183 6888 raft_consensus.cc:2932] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:19.571000 7278 raft_consensus.cc:2932] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:19.572068 6742 catalog_manager.cc:5484] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:19.595495 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:19.595464 9374 consensus_queue.cc:237] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.595650 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240503 07:59:19.597932 9310 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75f59b9fe4a54e7d8beabce7589800b1. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.598165 9767 raft_consensus.cc:1261] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.598543 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet dd7254c992bf4cbeb363b763d02e60a1
I20240503 07:59:19.598743 6888 raft_consensus.cc:1261] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.599193 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7254c992bf4cbeb363b763d02e60a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:19 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:19.599244 12145 consensus_queue.cc:1035] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.599596 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7254c992bf4cbeb363b763d02e60a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:19 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:19.599603 12145 consensus_queue.cc:1035] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.599798 12596 tablet_replica.cc:330] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:19.599931 12596 raft_consensus.cc:2229] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:19.599947 9746 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7254c992bf4cbeb363b763d02e60a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:19 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:19.600044 12596 raft_consensus.cc:2258] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:19.600919 12596 ts_tablet_manager.cc:1875] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:19.602682 12785 raft_consensus.cc:2932] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.602759 6888 raft_consensus.cc:2932] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.605227 12596 ts_tablet_manager.cc:1888] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:19.605340 12596 log.cc:1196] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/dd7254c992bf4cbeb363b763d02e60a1
I20240503 07:59:19.605600 12596 ts_tablet_manager.cc:1909] T dd7254c992bf4cbeb363b763d02e60a1 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:19.605672 9767 raft_consensus.cc:2932] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.606020 12787 tablet_replica.cc:330] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:19.606189 12787 raft_consensus.cc:2229] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:19.606402 12787 raft_consensus.cc:2258] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:19.606408 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet dd7254c992bf4cbeb363b763d02e60a1 (table rebalancer_dynamic_table_new_name [id=265b99f91f04467abc77330f48071a62]) successfully deleted
I20240503 07:59:19.607558 6740 catalog_manager.cc:5484] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 2, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.611584 12787 ts_tablet_manager.cc:1875] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:19.612645 12786 tablet_replica.cc:330] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:19.612820 12786 raft_consensus.cc:2229] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:19.613080 12786 raft_consensus.cc:2258] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:19.615092 12787 ts_tablet_manager.cc:1888] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:19.615170 12787 log.cc:1196] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/dd7254c992bf4cbeb363b763d02e60a1
I20240503 07:59:19.615419 12787 ts_tablet_manager.cc:1909] T dd7254c992bf4cbeb363b763d02e60a1 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:19.616247 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet dd7254c992bf4cbeb363b763d02e60a1 (table rebalancer_dynamic_table_new_name [id=265b99f91f04467abc77330f48071a62]) successfully deleted
I20240503 07:59:19.616235 12235 consensus_queue.cc:1035] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.616605 12786 ts_tablet_manager.cc:1875] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:19.619455 12786 ts_tablet_manager.cc:1888] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:19.619549 12786 log.cc:1196] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/dd7254c992bf4cbeb363b763d02e60a1
I20240503 07:59:19.619797 12786 ts_tablet_manager.cc:1909] T dd7254c992bf4cbeb363b763d02e60a1 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:19.620501 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet dd7254c992bf4cbeb363b763d02e60a1 (table rebalancer_dynamic_table_new_name [id=265b99f91f04467abc77330f48071a62]) successfully deleted
I20240503 07:59:19.620803 12235 consensus_queue.cc:1035] T 136fc500a47a4d9f98e48c5a0c7af17e P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.625767 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:19.632642 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 79dd3d8c050d4bba829de57fe39b025f (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2f64f0c37c8a48f68cd6e052fc7420fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.632920 6999 tablet_service.cc:1467] Processing CreateTablet for tablet be911252c4da43708b7ce6592d9b17f6 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2f64f0c37c8a48f68cd6e052fc7420fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.633031 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dd3d8c050d4bba829de57fe39b025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.633178 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be911252c4da43708b7ce6592d9b17f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.633687 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 79dd3d8c050d4bba829de57fe39b025f (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2f64f0c37c8a48f68cd6e052fc7420fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.633973 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dd3d8c050d4bba829de57fe39b025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.635892 9350 tablet_service.cc:1467] Processing CreateTablet for tablet a86b58fdebf942ab870d4b354414b8d2 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2f64f0c37c8a48f68cd6e052fc7420fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.636148 9353 tablet_service.cc:1467] Processing CreateTablet for tablet be911252c4da43708b7ce6592d9b17f6 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2f64f0c37c8a48f68cd6e052fc7420fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.636160 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86b58fdebf942ab870d4b354414b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.636415 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be911252c4da43708b7ce6592d9b17f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.636451 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 79dd3d8c050d4bba829de57fe39b025f (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2f64f0c37c8a48f68cd6e052fc7420fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.636703 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dd3d8c050d4bba829de57fe39b025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.638664 12807 tablet_bootstrap.cc:492] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:19.639169 9484 tablet_service.cc:1467] Processing CreateTablet for tablet be911252c4da43708b7ce6592d9b17f6 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2f64f0c37c8a48f68cd6e052fc7420fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.639405 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be911252c4da43708b7ce6592d9b17f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.640174 12807 tablet_bootstrap.cc:654] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.642105 6867 tablet_service.cc:1467] Processing CreateTablet for tablet a86b58fdebf942ab870d4b354414b8d2 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2f64f0c37c8a48f68cd6e052fc7420fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.642266 9615 tablet_service.cc:1467] Processing CreateTablet for tablet a86b58fdebf942ab870d4b354414b8d2 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2f64f0c37c8a48f68cd6e052fc7420fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.642405 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86b58fdebf942ab870d4b354414b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.642562 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86b58fdebf942ab870d4b354414b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.644734 12815 tablet_bootstrap.cc:492] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:19.646111 12804 tablet_bootstrap.cc:492] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:19.646121 9504 consensus_queue.cc:237] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:19.646329 12815 tablet_bootstrap.cc:654] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.646693 12808 tablet_bootstrap.cc:492] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:19.648043 12815 tablet_bootstrap.cc:492] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:19.648152 12815 ts_tablet_manager.cc:1367] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
W20240503 07:59:19.648332 9441 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): Couldn't send request to peer 5227b26243484bbfb177a995ceb50afc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df569029c1514bc5ae4d77327cd03478. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.648631 12815 raft_consensus.cc:348] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.648742 12815 raft_consensus.cc:374] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.648775 12815 raft_consensus.cc:729] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.648895 12815 consensus_queue.cc:260] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.649113 12805 tablet_bootstrap.cc:492] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:19.649191 12815 ts_tablet_manager.cc:1398] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:19.649762 9374 raft_consensus.cc:1261] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.649794 9636 raft_consensus.cc:1261] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.650238 12804 tablet_bootstrap.cc:654] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.650272 12235 consensus_queue.cc:1035] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.650641 12235 consensus_queue.cc:1035] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.652869 12805 tablet_bootstrap.cc:654] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.653452 12808 tablet_bootstrap.cc:654] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.653558 12806 tablet_bootstrap.cc:492] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:19.654520 12805 tablet_bootstrap.cc:492] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:19.654624 12805 ts_tablet_manager.cc:1367] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:19.654664 12808 tablet_bootstrap.cc:492] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:19.654760 12808 ts_tablet_manager.cc:1367] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:19.654768 12806 tablet_bootstrap.cc:654] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.655287 12805 raft_consensus.cc:348] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:19.655350 12808 raft_consensus.cc:348] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.655478 12805 raft_consensus.cc:374] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.655573 12805 raft_consensus.cc:729] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.655704 12805 consensus_queue.cc:260] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:19.655526 12808 raft_consensus.cc:374] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.655901 12808 raft_consensus.cc:729] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.656059 12808 consensus_queue.cc:260] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.656447 12808 ts_tablet_manager.cc:1398] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:19.656704 12808 tablet_bootstrap.cc:492] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:19.656787 12805 ts_tablet_manager.cc:1398] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:19.658536 12804 tablet_bootstrap.cc:492] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:19.658650 12804 ts_tablet_manager.cc:1367] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:19.659358 12804 raft_consensus.cc:348] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.659526 12804 raft_consensus.cc:374] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.659587 12804 raft_consensus.cc:729] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.659812 12804 consensus_queue.cc:260] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.660996 12804 ts_tablet_manager.cc:1398] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.661219 12804 tablet_bootstrap.cc:492] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:19.661401 11868 raft_consensus.cc:2932] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:19.662271 12808 tablet_bootstrap.cc:654] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.662781 12804 tablet_bootstrap.cc:654] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.664129 12807 tablet_bootstrap.cc:492] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:19.664228 12807 ts_tablet_manager.cc:1367] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 07:59:19.664315 6739 catalog_manager.cc:5484] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:19.664705 12807 raft_consensus.cc:348] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.664806 12807 raft_consensus.cc:374] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.664834 12807 raft_consensus.cc:729] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.664956 12807 consensus_queue.cc:260] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.665218 12807 ts_tablet_manager.cc:1398] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.665374 12807 tablet_bootstrap.cc:492] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:19.666287 12807 tablet_bootstrap.cc:654] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.666538 9374 raft_consensus.cc:2932] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:19.668754 9636 raft_consensus.cc:2932] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:19.669119 12806 tablet_bootstrap.cc:492] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:19.669235 12806 ts_tablet_manager.cc:1367] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 07:59:19.669759 12806 raft_consensus.cc:348] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.669876 12806 raft_consensus.cc:374] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.669907 12806 raft_consensus.cc:729] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.670049 12806 consensus_queue.cc:260] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.670526 12806 ts_tablet_manager.cc:1398] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.670980 12807 tablet_bootstrap.cc:492] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:19.671056 12807 ts_tablet_manager.cc:1367] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:19.671489 12807 raft_consensus.cc:348] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:19.671588 12807 raft_consensus.cc:374] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.671613 12807 raft_consensus.cc:729] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.671717 12807 consensus_queue.cc:260] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:19.671991 12807 ts_tablet_manager.cc:1398] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.673134 12804 tablet_bootstrap.cc:492] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:19.673401 12804 ts_tablet_manager.cc:1367] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:19.673885 12804 raft_consensus.cc:348] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.674047 12804 raft_consensus.cc:374] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.674145 12804 raft_consensus.cc:729] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.674335 12804 consensus_queue.cc:260] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.674888 12804 ts_tablet_manager.cc:1398] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:19.678692 12808 tablet_bootstrap.cc:492] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:19.678813 12808 ts_tablet_manager.cc:1367] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.002s
I20240503 07:59:19.679258 12808 raft_consensus.cc:348] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:19.679394 12808 raft_consensus.cc:374] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.679426 12808 raft_consensus.cc:729] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.679558 12808 consensus_queue.cc:260] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:19.679850 12808 ts_tablet_manager.cc:1398] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.699419 6888 consensus_queue.cc:237] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:19.700949 7020 raft_consensus.cc:1261] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:19.700953 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:19.701611 12525 consensus_queue.cc:1035] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:19.702497 7150 raft_consensus.cc:1261] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:19.702661 6825 consensus_peers.cc:485] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 -> Peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Couldn't send request to peer a372e98a670f49e9adca46ab5c836c4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3658573037664f03b38fefd31d2fe587. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.702911 12525 consensus_queue.cc:1035] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:19.705044 12729 raft_consensus.cc:2932] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:19.705437 7150 raft_consensus.cc:2932] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:19.707142 9615 tablet_service.cc:1467] Processing CreateTablet for tablet cadcaf386279483c92dbe1c518afbffe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=290c5a4b4eac476899647fb148506f22]), partition=
I20240503 07:59:19.707417 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadcaf386279483c92dbe1c518afbffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.707540 7020 raft_consensus.cc:2932] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:19.709334 12815 tablet_bootstrap.cc:492] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:19.710283 12815 tablet_bootstrap.cc:654] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.710325 6740 catalog_manager.cc:5484] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:19.711042 7129 tablet_service.cc:1467] Processing CreateTablet for tablet cadcaf386279483c92dbe1c518afbffe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=290c5a4b4eac476899647fb148506f22]), partition=
I20240503 07:59:19.711323 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadcaf386279483c92dbe1c518afbffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.711360 9484 tablet_service.cc:1467] Processing CreateTablet for tablet cadcaf386279483c92dbe1c518afbffe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=290c5a4b4eac476899647fb148506f22]), partition=
I20240503 07:59:19.711616 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadcaf386279483c92dbe1c518afbffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.713814 12806 tablet_bootstrap.cc:492] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:19.713814 12808 tablet_bootstrap.cc:492] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:19.713936 12815 tablet_bootstrap.cc:492] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:19.714051 12815 ts_tablet_manager.cc:1367] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:19.714625 12815 raft_consensus.cc:348] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.714712 12815 raft_consensus.cc:374] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.714733 12815 raft_consensus.cc:729] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.714825 12806 tablet_bootstrap.cc:654] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.714843 12815 consensus_queue.cc:260] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.714936 12808 tablet_bootstrap.cc:654] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.715091 12815 ts_tablet_manager.cc:1398] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.717028 12806 tablet_bootstrap.cc:492] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:19.717125 12806 ts_tablet_manager.cc:1367] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:19.718227 12806 raft_consensus.cc:348] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.718371 12806 raft_consensus.cc:374] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.718405 12806 raft_consensus.cc:729] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.718533 12806 consensus_queue.cc:260] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.718839 12806 ts_tablet_manager.cc:1398] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:19.719770 12808 tablet_bootstrap.cc:492] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:19.719868 12808 ts_tablet_manager.cc:1367] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:19.720265 12808 raft_consensus.cc:348] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.720361 12808 raft_consensus.cc:374] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.720392 12808 raft_consensus.cc:729] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.720523 12808 consensus_queue.cc:260] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.720769 12808 ts_tablet_manager.cc:1398] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.720963 11512 raft_consensus.cc:482] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.721057 11512 raft_consensus.cc:504] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.721508 11512 leader_election.cc:288] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:19.721910 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadcaf386279483c92dbe1c518afbffe" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:19.722038 7150 raft_consensus.cc:2454] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:19.722102 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadcaf386279483c92dbe1c518afbffe" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:19.722222 9504 raft_consensus.cc:2454] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:19.722375 9572 leader_election.cc:302] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2; no voters:
I20240503 07:59:19.722524 11512 raft_consensus.cc:2781] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.722569 11512 raft_consensus.cc:482] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.722609 11512 raft_consensus.cc:3037] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.723415 11512 raft_consensus.cc:504] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.723766 11512 leader_election.cc:288] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:19.724102 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadcaf386279483c92dbe1c518afbffe" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:19.724129 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadcaf386279483c92dbe1c518afbffe" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:19.724191 7150 raft_consensus.cc:3037] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.724231 9504 raft_consensus.cc:3037] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.724952 7150 raft_consensus.cc:2454] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:19.725196 9504 raft_consensus.cc:2454] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:19.725224 9572 leader_election.cc:302] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2; no voters:
I20240503 07:59:19.725391 11512 raft_consensus.cc:2781] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.725467 11512 raft_consensus.cc:686] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:19.725598 11512 consensus_queue.cc:237] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.727427 6738 catalog_manager.cc:5484] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.791659 12790 raft_consensus.cc:482] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.791805 12790 raft_consensus.cc:504] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.792271 12790 leader_election.cc:288] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:19.792685 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86b58fdebf942ab870d4b354414b8d2" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:19.792718 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86b58fdebf942ab870d4b354414b8d2" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:19.792840 9636 raft_consensus.cc:2454] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:19.792888 6888 raft_consensus.cc:2454] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:19.793170 9307 leader_election.cc:302] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:19.793522 12790 raft_consensus.cc:2781] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.793579 12790 raft_consensus.cc:482] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.793615 12790 raft_consensus.cc:3037] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.794720 12790 raft_consensus.cc:504] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.795063 12790 leader_election.cc:288] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:19.795799 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86b58fdebf942ab870d4b354414b8d2" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:19.795936 6888 raft_consensus.cc:3037] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.797044 6888 raft_consensus.cc:2454] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:19.797504 9307 leader_election.cc:302] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:19.797808 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86b58fdebf942ab870d4b354414b8d2" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:19.797926 9636 raft_consensus.cc:3037] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.798978 9636 raft_consensus.cc:2454] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:19.800266 12790 raft_consensus.cc:2781] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.800369 12790 raft_consensus.cc:686] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:19.800508 12790 consensus_queue.cc:237] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:19.803212 6741 catalog_manager.cc:5484] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:19.830173 12893 ts_tablet_manager.cc:902] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:19.830689 12893 tablet_copy_client.cc:308] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:19.831235 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 7c2386cd009847d9a1fe7a959f47086c from peer a372e98a670f49e9adca46ab5c836c4a ({username='slave'} at 127.0.99.136:53081)
I20240503 07:59:19.831357 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 7c2386cd009847d9a1fe7a959f47086c from peer a372e98a670f49e9adca46ab5c836c4a at {username='slave'} at 127.0.99.136:53081: session id = a372e98a670f49e9adca46ab5c836c4a-7c2386cd009847d9a1fe7a959f47086c
I20240503 07:59:19.831462 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:19.832512 7171 tablet_copy_source_session.cc:205] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.832901 12898 ts_tablet_manager.cc:902] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:19.833516 12893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2386cd009847d9a1fe7a959f47086c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.833827 12898 tablet_copy_client.cc:308] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:19.834424 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet c27273ae45c449018143cc55e65336a6 from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:48617)
I20240503 07:59:19.834524 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet c27273ae45c449018143cc55e65336a6 from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:48617: session id = cb73994515af4880b05383e78368bb4a-c27273ae45c449018143cc55e65336a6
I20240503 07:59:19.835275 7171 tablet_copy_source_session.cc:205] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.836112 12893 tablet_copy_client.cc:783] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.836274 12893 tablet_copy_client.cc:647] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.836681 12898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27273ae45c449018143cc55e65336a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.839221 12893 tablet_copy_client.cc:523] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.839337 9504 raft_consensus.cc:1261] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.839378 7150 raft_consensus.cc:1261] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.839888 12669 consensus_queue.cc:1035] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.840224 12669 consensus_queue.cc:1035] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.841756 12898 tablet_copy_client.cc:783] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.841928 12898 tablet_copy_client.cc:647] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.843488 6739 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet cadcaf386279483c92dbe1c518afbffe (table rebalancer_dynamic_table [id=290c5a4b4eac476899647fb148506f22]). Expected version 1 got 0
I20240503 07:59:19.843845 6741 catalog_manager.cc:5576] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317) does not have the latest schema for tablet cadcaf386279483c92dbe1c518afbffe (table rebalancer_dynamic_table [id=290c5a4b4eac476899647fb148506f22]). Expected version 1 got 0
I20240503 07:59:19.843983 12580 tablet.cc:1722] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:19.844290 12898 tablet_copy_client.cc:523] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.844369 12580 tablet.cc:1620] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:19.844667 12578 tablet.cc:1722] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:19.844955 12578 tablet.cc:1620] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:19.846457 12898 tablet_bootstrap.cc:492] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:19.848537 12893 tablet_bootstrap.cc:492] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:19.848446 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=290c5a4b4eac476899647fb148506f22] alter complete (version 1)
I20240503 07:59:19.855245 12908 tablet.cc:1722] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:19.855595 12908 tablet.cc:1620] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:19.856439 6739 catalog_manager.cc:5576] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317) does not have the latest schema for tablet cadcaf386279483c92dbe1c518afbffe (table rebalancer_dynamic_table [id=290c5a4b4eac476899647fb148506f22]). Expected version 1 got 0
I20240503 07:59:19.876771 12893 tablet_bootstrap.cc:492] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.877177 12893 tablet_bootstrap.cc:492] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a: Bootstrap complete.
I20240503 07:59:19.877386 12893 ts_tablet_manager.cc:1367] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.000s
I20240503 07:59:19.877887 12893 raft_consensus.cc:348] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:19.878029 12893 raft_consensus.cc:729] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.878193 12893 consensus_queue.cc:260] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:19.878649 12893 ts_tablet_manager.cc:1398] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:19.879318 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session a372e98a670f49e9adca46ab5c836c4a-7c2386cd009847d9a1fe7a959f47086c received from {username='slave'} at 127.0.99.136:53081
I20240503 07:59:19.879406 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session a372e98a670f49e9adca46ab5c836c4a-7c2386cd009847d9a1fe7a959f47086c on tablet 7c2386cd009847d9a1fe7a959f47086c with peer a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:19.880447 12898 tablet_bootstrap.cc:492] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.880743 12898 tablet_bootstrap.cc:492] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:19.880950 12898 ts_tablet_manager.cc:1367] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.000s
I20240503 07:59:19.881402 12898 raft_consensus.cc:348] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.881513 12898 raft_consensus.cc:729] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.881655 12898 consensus_queue.cc:260] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.881959 12898 ts_tablet_manager.cc:1398] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:59:19.882622 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-c27273ae45c449018143cc55e65336a6 received from {username='slave'} at 127.0.99.134:48617
I20240503 07:59:19.882719 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session cb73994515af4880b05383e78368bb4a-c27273ae45c449018143cc55e65336a6 on tablet c27273ae45c449018143cc55e65336a6 with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:19.885794 12790 raft_consensus.cc:482] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.885924 12790 raft_consensus.cc:504] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:19.886502 12790 leader_election.cc:288] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:19.886870 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be911252c4da43708b7ce6592d9b17f6" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:19.886921 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be911252c4da43708b7ce6592d9b17f6" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:19.887017 9504 raft_consensus.cc:2454] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:19.887048 7020 raft_consensus.cc:2454] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:19.887312 9310 leader_election.cc:302] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:19.887521 12790 raft_consensus.cc:2781] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.887584 12790 raft_consensus.cc:482] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.887652 12790 raft_consensus.cc:3037] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.888626 12790 raft_consensus.cc:504] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:19.889030 12790 leader_election.cc:288] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:19.889338 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be911252c4da43708b7ce6592d9b17f6" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:19.889448 7020 raft_consensus.cc:3037] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.889446 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be911252c4da43708b7ce6592d9b17f6" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:19.889546 9504 raft_consensus.cc:3037] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.890416 9504 raft_consensus.cc:2454] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:19.890419 7020 raft_consensus.cc:2454] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:19.890743 9310 leader_election.cc:302] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:19.890976 12790 raft_consensus.cc:2781] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.891103 12790 raft_consensus.cc:686] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:19.891260 12790 consensus_queue.cc:237] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:19.893198 6741 catalog_manager.cc:5484] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:19.894902 12320 raft_consensus.cc:482] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.895004 12320 raft_consensus.cc:504] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.895509 12320 leader_election.cc:288] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:19.895928 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dd3d8c050d4bba829de57fe39b025f" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:19.895928 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dd3d8c050d4bba829de57fe39b025f" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:19.896071 6888 raft_consensus.cc:2454] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:19.896071 9504 raft_consensus.cc:2454] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:19.896353 7087 leader_election.cc:302] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:19.896551 12320 raft_consensus.cc:2781] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.896615 12320 raft_consensus.cc:482] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.896664 12320 raft_consensus.cc:3037] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.897668 12320 raft_consensus.cc:504] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.898038 12320 leader_election.cc:288] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:19.898408 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dd3d8c050d4bba829de57fe39b025f" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:19.898425 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dd3d8c050d4bba829de57fe39b025f" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:19.898519 6888 raft_consensus.cc:3037] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.898528 9504 raft_consensus.cc:3037] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.899267 6888 raft_consensus.cc:2454] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:19.899271 9504 raft_consensus.cc:2454] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:19.899569 7084 leader_election.cc:302] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:19.899767 12320 raft_consensus.cc:2781] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.899881 12320 raft_consensus.cc:686] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:19.900033 12320 consensus_queue.cc:237] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:19.901942 6741 catalog_manager.cc:5484] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:19.920413 9353 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:19.920516 7129 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:19.925135 9615 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:19.935994 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:19.937646 12898 ts_tablet_manager.cc:902] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:19.937701 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:19.938114 12898 tablet_copy_client.cc:308] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:19.938874 9746 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:19.939548 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet aa45918bb38c4583ad59524bda2d48ee from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:48369)
I20240503 07:59:19.939666 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet aa45918bb38c4583ad59524bda2d48ee from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:48369: session id = cb73994515af4880b05383e78368bb4a-aa45918bb38c4583ad59524bda2d48ee
I20240503 07:59:19.940762 7302 tablet_copy_source_session.cc:205] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.942930 9484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:19.943558 12898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa45918bb38c4583ad59524bda2d48ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.946081 12898 tablet_copy_client.cc:783] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.946252 12898 tablet_copy_client.cc:647] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.949002 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:19.951783 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:19.953181 12898 tablet_copy_client.cc:523] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.962473 12898 tablet_bootstrap.cc:492] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:19.964427 12580 tablet.cc:1722] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:19.964774 12580 tablet.cc:1620] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:19.971707 12928 ts_tablet_manager.cc:902] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:19.972287 12928 tablet_copy_client.cc:308] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46383
I20240503 07:59:19.972947 7039 tablet_copy_service.cc:140] P 9c085a3241f14cb7ba1332b508f4b57d: Received BeginTabletCopySession request for tablet ea625a1834fb4a038a32a90d2fcbf1ac from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:49881)
I20240503 07:59:19.973052 7039 tablet_copy_service.cc:161] P 9c085a3241f14cb7ba1332b508f4b57d: Beginning new tablet copy session on tablet ea625a1834fb4a038a32a90d2fcbf1ac from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:49881: session id = 159177b3caeb434baa98e04136d050b2-ea625a1834fb4a038a32a90d2fcbf1ac
I20240503 07:59:19.973935 12578 tablet.cc:1722] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:19.974066 7039 tablet_copy_source_session.cc:205] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.974278 12578 tablet.cc:1620] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:19.974437 12908 tablet.cc:1722] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:19.974725 12908 tablet.cc:1620] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:19.978232 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=290c5a4b4eac476899647fb148506f22] alter complete (version 2)
I20240503 07:59:19.987836 12928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea625a1834fb4a038a32a90d2fcbf1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.992569 12928 tablet_copy_client.cc:783] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.994552 12928 tablet_copy_client.cc:647] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.995981 12928 tablet_copy_client.cc:523] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.997530 12928 tablet_bootstrap.cc:492] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:20.005395 12933 ts_tablet_manager.cc:902] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:20.006245 12933 tablet_copy_client.cc:308] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:20.007009 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 5a838dc6e3114be5aceac51d567bc645 from peer 9c085a3241f14cb7ba1332b508f4b57d ({username='slave'} at 127.0.99.130:45195)
I20240503 07:59:20.007148 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 5a838dc6e3114be5aceac51d567bc645 from peer 9c085a3241f14cb7ba1332b508f4b57d at {username='slave'} at 127.0.99.130:45195: session id = 9c085a3241f14cb7ba1332b508f4b57d-5a838dc6e3114be5aceac51d567bc645
I20240503 07:59:20.008244 7171 tablet_copy_source_session.cc:205] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:20.010754 12933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a838dc6e3114be5aceac51d567bc645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.016603 12933 tablet_copy_client.cc:783] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:20.016839 12933 tablet_copy_client.cc:647] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:20.018864 12933 tablet_copy_client.cc:523] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:20.020977 12933 tablet_bootstrap.cc:492] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:20.032959 12933 tablet_bootstrap.cc:492] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:20.033329 12933 tablet_bootstrap.cc:492] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap complete.
I20240503 07:59:20.033548 12933 ts_tablet_manager.cc:1367] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240503 07:59:20.034152 12933 raft_consensus.cc:348] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:20.034371 12933 raft_consensus.cc:729] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.034538 12933 consensus_queue.cc:260] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:20.034999 12933 ts_tablet_manager.cc:1398] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.036000 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-5a838dc6e3114be5aceac51d567bc645 received from {username='slave'} at 127.0.99.130:45195
I20240503 07:59:20.036100 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-5a838dc6e3114be5aceac51d567bc645 on tablet 5a838dc6e3114be5aceac51d567bc645 with peer 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:20.041922 12928 tablet_bootstrap.cc:492] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:20.042407 12928 tablet_bootstrap.cc:492] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:20.042649 12928 ts_tablet_manager.cc:1367] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.045s user 0.028s sys 0.001s
I20240503 07:59:20.043116 12947 ts_tablet_manager.cc:902] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc: Initiating tablet copy from peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:20.043143 12928 raft_consensus.cc:348] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:20.043267 12928 raft_consensus.cc:729] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.043408 12928 consensus_queue.cc:260] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:20.044498 12928 ts_tablet_manager.cc:1398] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:20.044936 7039 tablet_copy_service.cc:342] P 9c085a3241f14cb7ba1332b508f4b57d: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-ea625a1834fb4a038a32a90d2fcbf1ac received from {username='slave'} at 127.0.99.135:49881
I20240503 07:59:20.045020 7039 tablet_copy_service.cc:433] P 9c085a3241f14cb7ba1332b508f4b57d: ending tablet copy session 159177b3caeb434baa98e04136d050b2-ea625a1834fb4a038a32a90d2fcbf1ac on tablet ea625a1834fb4a038a32a90d2fcbf1ac with peer 159177b3caeb434baa98e04136d050b2
I20240503 07:59:20.046130 12947 tablet_copy_client.cc:308] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:45435
I20240503 07:59:20.048271 9525 tablet_copy_service.cc:140] P cb73994515af4880b05383e78368bb4a: Received BeginTabletCopySession request for tablet df569029c1514bc5ae4d77327cd03478 from peer 5227b26243484bbfb177a995ceb50afc ({username='slave'} at 127.0.99.132:35461)
I20240503 07:59:20.048388 9525 tablet_copy_service.cc:161] P cb73994515af4880b05383e78368bb4a: Beginning new tablet copy session on tablet df569029c1514bc5ae4d77327cd03478 from peer 5227b26243484bbfb177a995ceb50afc at {username='slave'} at 127.0.99.132:35461: session id = 5227b26243484bbfb177a995ceb50afc-df569029c1514bc5ae4d77327cd03478
I20240503 07:59:20.048867 12898 tablet_bootstrap.cc:492] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:20.049130 12898 tablet_bootstrap.cc:492] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:20.049297 12898 ts_tablet_manager.cc:1367] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.087s user 0.025s sys 0.000s
I20240503 07:59:20.049412 9525 tablet_copy_source_session.cc:205] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:20.051012 12947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df569029c1514bc5ae4d77327cd03478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.051529 12898 raft_consensus.cc:348] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.051689 12898 raft_consensus.cc:729] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.051827 12898 consensus_queue.cc:260] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.052587 12898 ts_tablet_manager.cc:1398] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:20.053683 12947 tablet_copy_client.cc:783] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:20.053926 12947 tablet_copy_client.cc:647] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:20.054044 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-aa45918bb38c4583ad59524bda2d48ee received from {username='slave'} at 127.0.99.134:48369
I20240503 07:59:20.054181 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session cb73994515af4880b05383e78368bb4a-aa45918bb38c4583ad59524bda2d48ee on tablet aa45918bb38c4583ad59524bda2d48ee with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:20.059230 12947 tablet_copy_client.cc:523] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:20.060986 12947 tablet_bootstrap.cc:492] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:20.065596 12947 tablet_bootstrap.cc:492] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:20.065847 12947 tablet_bootstrap.cc:492] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc: Bootstrap complete.
I20240503 07:59:20.066004 12947 ts_tablet_manager.cc:1367] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:59:20.066483 12947 raft_consensus.cc:348] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:20.066588 12947 raft_consensus.cc:729] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.066735 12947 consensus_queue.cc:260] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:20.067003 12947 ts_tablet_manager.cc:1398] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:20.067641 9525 tablet_copy_service.cc:342] P cb73994515af4880b05383e78368bb4a: Request end of tablet copy session 5227b26243484bbfb177a995ceb50afc-df569029c1514bc5ae4d77327cd03478 received from {username='slave'} at 127.0.99.132:35461
I20240503 07:59:20.067732 9525 tablet_copy_service.cc:433] P cb73994515af4880b05383e78368bb4a: ending tablet copy session 5227b26243484bbfb177a995ceb50afc-df569029c1514bc5ae4d77327cd03478 on tablet df569029c1514bc5ae4d77327cd03478 with peer 5227b26243484bbfb177a995ceb50afc
I20240503 07:59:20.082971 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:20.084472 12898 ts_tablet_manager.cc:902] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:20.084586 12893 ts_tablet_manager.cc:902] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:20.084921 12898 tablet_copy_client.cc:308] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37815
I20240503 07:59:20.085001 12893 tablet_copy_client.cc:308] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:20.085516 6908 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet 5b794508aa77492398e48d25edfb8aa3 from peer a372e98a670f49e9adca46ab5c836c4a ({username='slave'} at 127.0.99.136:39953)
I20240503 07:59:20.085638 6908 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet 5b794508aa77492398e48d25edfb8aa3 from peer a372e98a670f49e9adca46ab5c836c4a at {username='slave'} at 127.0.99.136:39953: session id = a372e98a670f49e9adca46ab5c836c4a-5b794508aa77492398e48d25edfb8aa3
I20240503 07:59:20.086908 9394 tablet_copy_service.cc:140] P 3befbb2d206f4be89d3e07f38837374b: Received BeginTabletCopySession request for tablet 75f59b9fe4a54e7d8beabce7589800b1 from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:44093)
I20240503 07:59:20.087034 9394 tablet_copy_service.cc:161] P 3befbb2d206f4be89d3e07f38837374b: Beginning new tablet copy session on tablet 75f59b9fe4a54e7d8beabce7589800b1 from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:44093: session id = cb73994515af4880b05383e78368bb4a-75f59b9fe4a54e7d8beabce7589800b1
I20240503 07:59:20.086735 6908 tablet_copy_source_session.cc:205] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:20.088723 9394 tablet_copy_source_session.cc:205] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:20.089561 12893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b794508aa77492398e48d25edfb8aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.094496 12898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f59b9fe4a54e7d8beabce7589800b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.095413 12893 tablet_copy_client.cc:783] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:20.095587 12893 tablet_copy_client.cc:647] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:20.099153 12580 tablet.cc:1722] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:20.099772 12578 tablet.cc:1722] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:20.101801 12908 tablet.cc:1722] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:20.103649 12898 tablet_copy_client.cc:783] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:20.103935 12898 tablet_copy_client.cc:647] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:20.106392 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=290c5a4b4eac476899647fb148506f22] alter complete (version 3)
I20240503 07:59:20.108060 12893 tablet_copy_client.cc:523] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:20.109833 12898 tablet_copy_client.cc:523] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:20.110373 12893 tablet_bootstrap.cc:492] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:20.112366 12898 tablet_bootstrap.cc:492] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:20.116582 12898 tablet_bootstrap.cc:492] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:20.116832 12898 tablet_bootstrap.cc:492] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:20.116984 12898 ts_tablet_manager.cc:1367] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:59:20.117439 12898 raft_consensus.cc:348] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.117563 12898 raft_consensus.cc:729] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.117789 12898 consensus_queue.cc:260] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.118185 12898 ts_tablet_manager.cc:1398] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.118809 9394 tablet_copy_service.cc:342] P 3befbb2d206f4be89d3e07f38837374b: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-75f59b9fe4a54e7d8beabce7589800b1 received from {username='slave'} at 127.0.99.134:44093
I20240503 07:59:20.118909 9394 tablet_copy_service.cc:433] P 3befbb2d206f4be89d3e07f38837374b: ending tablet copy session cb73994515af4880b05383e78368bb4a-75f59b9fe4a54e7d8beabce7589800b1 on tablet 75f59b9fe4a54e7d8beabce7589800b1 with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:20.129909 13010 ts_tablet_manager.cc:902] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:20.130609 13010 tablet_copy_client.cc:308] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:20.131125 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 795f69249afe444ba0828875e1839b8c from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:52953)
I20240503 07:59:20.131234 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 795f69249afe444ba0828875e1839b8c from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:52953: session id = 3befbb2d206f4be89d3e07f38837374b-795f69249afe444ba0828875e1839b8c
I20240503 07:59:20.132071 7171 tablet_copy_source_session.cc:205] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:20.133010 13010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795f69249afe444ba0828875e1839b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.135412 13010 tablet_copy_client.cc:783] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:20.135636 13010 tablet_copy_client.cc:647] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:20.135803 12893 tablet_bootstrap.cc:492] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:20.136073 12893 tablet_bootstrap.cc:492] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap complete.
I20240503 07:59:20.136257 12893 ts_tablet_manager.cc:1367] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.000s
I20240503 07:59:20.136723 12893 raft_consensus.cc:348] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.136844 12893 raft_consensus.cc:729] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.137017 12893 consensus_queue.cc:260] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.137310 12893 ts_tablet_manager.cc:1398] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:59:20.137838 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session a372e98a670f49e9adca46ab5c836c4a-5b794508aa77492398e48d25edfb8aa3 received from {username='slave'} at 127.0.99.136:39953
I20240503 07:59:20.137920 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session a372e98a670f49e9adca46ab5c836c4a-5b794508aa77492398e48d25edfb8aa3 on tablet 5b794508aa77492398e48d25edfb8aa3 with peer a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:20.138403 13010 tablet_copy_client.cc:523] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:20.140452 13010 tablet_bootstrap.cc:492] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:20.161963 13010 tablet_bootstrap.cc:492] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:20.162268 13010 tablet_bootstrap.cc:492] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:20.162467 13010 ts_tablet_manager.cc:1367] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.002s
I20240503 07:59:20.162967 13010 raft_consensus.cc:348] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:20.163183 13010 raft_consensus.cc:729] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.163398 13010 consensus_queue.cc:260] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:20.163813 13010 ts_tablet_manager.cc:1398] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.164391 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-795f69249afe444ba0828875e1839b8c received from {username='slave'} at 127.0.99.133:52953
I20240503 07:59:20.164470 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-795f69249afe444ba0828875e1839b8c on tablet 795f69249afe444ba0828875e1839b8c with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:20.181779 12893 ts_tablet_manager.cc:902] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:20.182336 12893 tablet_copy_client.cc:308] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:20.183176 6908 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet 3658573037664f03b38fefd31d2fe587 from peer a372e98a670f49e9adca46ab5c836c4a ({username='slave'} at 127.0.99.136:39953)
I20240503 07:59:20.183298 6908 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet 3658573037664f03b38fefd31d2fe587 from peer a372e98a670f49e9adca46ab5c836c4a at {username='slave'} at 127.0.99.136:39953: session id = a372e98a670f49e9adca46ab5c836c4a-3658573037664f03b38fefd31d2fe587
I20240503 07:59:20.184398 6908 tablet_copy_source_session.cc:205] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:20.186473 12893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3658573037664f03b38fefd31d2fe587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.189008 12893 tablet_copy_client.cc:783] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:20.189182 12893 tablet_copy_client.cc:647] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:20.190531 12893 tablet_copy_client.cc:523] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:20.192004 12893 tablet_bootstrap.cc:492] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:20.210716 6747 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:20.210870 6747 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:20.213269 6747 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet cadcaf386279483c92dbe1c518afbffe
I20240503 07:59:20.214030 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet cadcaf386279483c92dbe1c518afbffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:20 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:20.215803 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet cadcaf386279483c92dbe1c518afbffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:20 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:20.218436 9484 tablet_service.cc:1514] Processing DeleteTablet for tablet cadcaf386279483c92dbe1c518afbffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:20 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:20.218755 13064 tablet_replica.cc:330] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:20.218976 13064 raft_consensus.cc:2229] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.219214 13064 raft_consensus.cc:2258] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.220027 13064 ts_tablet_manager.cc:1875] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:20.222573 12893 tablet_bootstrap.cc:492] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:20.222607 13064 ts_tablet_manager.cc:1888] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:20.222826 13064 log.cc:1196] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/cadcaf386279483c92dbe1c518afbffe
I20240503 07:59:20.222858 12893 tablet_bootstrap.cc:492] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap complete.
I20240503 07:59:20.223018 12893 ts_tablet_manager.cc:1367] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.000s
I20240503 07:59:20.223361 13064 ts_tablet_manager.cc:1909] T cadcaf386279483c92dbe1c518afbffe P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:20.223474 12893 raft_consensus.cc:348] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.223593 12893 raft_consensus.cc:729] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.223783 7129 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:20.223773 12893 consensus_queue.cc:260] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.224054 12893 ts_tablet_manager.cc:1398] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.224668 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet cadcaf386279483c92dbe1c518afbffe (table rebalancer_dynamic_table_new_name [id=290c5a4b4eac476899647fb148506f22]) successfully deleted
I20240503 07:59:20.224948 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session a372e98a670f49e9adca46ab5c836c4a-3658573037664f03b38fefd31d2fe587 received from {username='slave'} at 127.0.99.136:39953
I20240503 07:59:20.225025 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session a372e98a670f49e9adca46ab5c836c4a-3658573037664f03b38fefd31d2fe587 on tablet 3658573037664f03b38fefd31d2fe587 with peer a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:20.228154 13062 tablet_replica.cc:330] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:20.234799 13062 raft_consensus.cc:2229] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:20.235102 13062 raft_consensus.cc:2258] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.235796 13062 ts_tablet_manager.cc:1875] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:20.238265 13062 ts_tablet_manager.cc:1888] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:20.238363 13062 log.cc:1196] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/cadcaf386279483c92dbe1c518afbffe
I20240503 07:59:20.238611 13062 ts_tablet_manager.cc:1909] T cadcaf386279483c92dbe1c518afbffe P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:20.239784 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet cadcaf386279483c92dbe1c518afbffe (table rebalancer_dynamic_table_new_name [id=290c5a4b4eac476899647fb148506f22]) successfully deleted
I20240503 07:59:20.249397 13063 tablet_replica.cc:330] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:20.252513 9615 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:20.253777 13063 raft_consensus.cc:2229] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.253993 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:20.254043 13063 raft_consensus.cc:2258] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.256039 13063 ts_tablet_manager.cc:1875] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:20.256791 9484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:20.259049 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:20.259335 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:20.260293 13063 ts_tablet_manager.cc:1888] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:20.262023 13063 log.cc:1196] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/cadcaf386279483c92dbe1c518afbffe
I20240503 07:59:20.262495 13063 ts_tablet_manager.cc:1909] T cadcaf386279483c92dbe1c518afbffe P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:20.263830 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet cadcaf386279483c92dbe1c518afbffe (table rebalancer_dynamic_table_new_name [id=290c5a4b4eac476899647fb148506f22]) successfully deleted
I20240503 07:59:20.281747 9746 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:20.295011 9353 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:20.309082 9767 raft_consensus.cc:1203] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:20.315191 6747 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:20.321554 9353 tablet_service.cc:1467] Processing CreateTablet for tablet 166a5a44983c43d581c0332878cebfdd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=295d54d12ba2425d855dd1fa0513f8f7]), partition=
I20240503 07:59:20.321852 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166a5a44983c43d581c0332878cebfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.322631 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 166a5a44983c43d581c0332878cebfdd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=295d54d12ba2425d855dd1fa0513f8f7]), partition=
I20240503 07:59:20.322886 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166a5a44983c43d581c0332878cebfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.324371 13066 tablet_bootstrap.cc:492] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:20.325134 13067 tablet_bootstrap.cc:492] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:20.325264 9484 tablet_service.cc:1467] Processing CreateTablet for tablet 166a5a44983c43d581c0332878cebfdd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=295d54d12ba2425d855dd1fa0513f8f7]), partition=
I20240503 07:59:20.325511 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166a5a44983c43d581c0332878cebfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.325654 13066 tablet_bootstrap.cc:654] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.326205 13067 tablet_bootstrap.cc:654] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.327791 13070 tablet_bootstrap.cc:492] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:20.328899 13070 tablet_bootstrap.cc:654] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.329389 13067 tablet_bootstrap.cc:492] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:20.329537 13067 ts_tablet_manager.cc:1367] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:20.330101 13070 tablet_bootstrap.cc:492] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:20.330086 13067 raft_consensus.cc:348] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.330199 13067 raft_consensus.cc:374] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.330229 13067 raft_consensus.cc:729] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.330268 13070 ts_tablet_manager.cc:1367] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:20.330415 13067 consensus_queue.cc:260] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.330720 13067 ts_tablet_manager.cc:1398] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:20.330941 13070 raft_consensus.cc:348] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.331051 13070 raft_consensus.cc:374] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.331077 13070 raft_consensus.cc:729] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.331197 13070 consensus_queue.cc:260] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.333040 13070 ts_tablet_manager.cc:1398] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:20.337922 13066 tablet_bootstrap.cc:492] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:20.338140 13066 ts_tablet_manager.cc:1367] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:20.338757 13066 raft_consensus.cc:348] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.338963 13066 raft_consensus.cc:374] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.339044 13066 raft_consensus.cc:729] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.339208 13066 consensus_queue.cc:260] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.339687 13066 ts_tablet_manager.cc:1398] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.345444 12790 consensus_queue.cc:1035] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.348649 12790 consensus_queue.cc:1035] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.361052 12790 raft_consensus.cc:482] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:20.361208 12790 raft_consensus.cc:504] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.361608 12790 leader_election.cc:288] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:20.362212 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166a5a44983c43d581c0332878cebfdd" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:20.362383 9767 raft_consensus.cc:2454] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:20.362663 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166a5a44983c43d581c0332878cebfdd" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:20.362700 9310 leader_election.cc:302] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:20.362793 9504 raft_consensus.cc:2454] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:20.362874 12790 raft_consensus.cc:2781] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:20.362927 12790 raft_consensus.cc:482] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:20.362963 12790 raft_consensus.cc:3037] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.364025 12790 raft_consensus.cc:504] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.364396 12790 leader_election.cc:288] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:20.364812 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166a5a44983c43d581c0332878cebfdd" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:20.364941 9767 raft_consensus.cc:3037] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.365938 9767 raft_consensus.cc:2454] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:20.366202 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166a5a44983c43d581c0332878cebfdd" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:20.366245 9310 leader_election.cc:302] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:20.366323 9504 raft_consensus.cc:3037] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.366451 12790 raft_consensus.cc:2781] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.366531 12790 raft_consensus.cc:686] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:20.366667 12790 consensus_queue.cc:237] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.367343 9504 raft_consensus.cc:2454] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:20.368865 6747 catalog_manager.cc:5484] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:20.373714 12320 consensus_queue.cc:1035] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.378044 12644 consensus_queue.cc:1035] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.416311 9634 raft_consensus.cc:1203] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:20.421217 12790 consensus_queue.cc:1035] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.427284 12785 consensus_queue.cc:1035] T be911252c4da43708b7ce6592d9b17f6 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.438513 9504 raft_consensus.cc:1203] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:20.448776 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:20.459744 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 21f55eb986bd4b7bbda10e683f3bfcad (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=52f5ffbd2adb439692c9dea927b73abb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.460114 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21f55eb986bd4b7bbda10e683f3bfcad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.460068 9746 tablet_service.cc:1467] Processing CreateTablet for tablet fbc7505d3ef6416c9d505bfe899f73ed (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=52f5ffbd2adb439692c9dea927b73abb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.460333 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc7505d3ef6416c9d505bfe899f73ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.460594 9615 tablet_service.cc:1467] Processing CreateTablet for tablet b607148b2a7b4b56b83e9606be49221b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=52f5ffbd2adb439692c9dea927b73abb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.460835 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b607148b2a7b4b56b83e9606be49221b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.462507 13066 tablet_bootstrap.cc:492] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:20.462635 9484 tablet_service.cc:1467] Processing CreateTablet for tablet fbc7505d3ef6416c9d505bfe899f73ed (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=52f5ffbd2adb439692c9dea927b73abb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.462641 9351 tablet_service.cc:1467] Processing CreateTablet for tablet b607148b2a7b4b56b83e9606be49221b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=52f5ffbd2adb439692c9dea927b73abb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.462894 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc7505d3ef6416c9d505bfe899f73ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.462893 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b607148b2a7b4b56b83e9606be49221b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.463894 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 21f55eb986bd4b7bbda10e683f3bfcad (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=52f5ffbd2adb439692c9dea927b73abb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.464186 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21f55eb986bd4b7bbda10e683f3bfcad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.464226 13066 tablet_bootstrap.cc:654] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.464877 13067 tablet_bootstrap.cc:492] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:20.465262 7131 tablet_service.cc:1467] Processing CreateTablet for tablet b607148b2a7b4b56b83e9606be49221b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=52f5ffbd2adb439692c9dea927b73abb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.465482 13070 tablet_bootstrap.cc:492] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:20.465517 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b607148b2a7b4b56b83e9606be49221b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.465866 13067 tablet_bootstrap.cc:654] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.459744 9353 tablet_service.cc:1467] Processing CreateTablet for tablet fbc7505d3ef6416c9d505bfe899f73ed (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=52f5ffbd2adb439692c9dea927b73abb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.466858 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc7505d3ef6416c9d505bfe899f73ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.468732 13067 tablet_bootstrap.cc:492] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:20.468839 13067 ts_tablet_manager.cc:1367] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 07:59:20.469379 13067 raft_consensus.cc:348] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.469501 13067 raft_consensus.cc:374] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.469533 13067 raft_consensus.cc:729] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.470324 13070 tablet_bootstrap.cc:654] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.470412 13067 consensus_queue.cc:260] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.472467 13067 ts_tablet_manager.cc:1398] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:20.474999 7131 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:20.475796 9615 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:20.477651 9484 tablet_service.cc:1467] Processing CreateTablet for tablet 21f55eb986bd4b7bbda10e683f3bfcad (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=52f5ffbd2adb439692c9dea927b73abb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.477970 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21f55eb986bd4b7bbda10e683f3bfcad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.480190 9350 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:20.481596 13066 tablet_bootstrap.cc:492] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:20.481734 13066 ts_tablet_manager.cc:1367] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240503 07:59:20.482262 13066 raft_consensus.cc:348] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.482415 13066 raft_consensus.cc:374] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.482450 13066 raft_consensus.cc:729] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.482568 13066 consensus_queue.cc:260] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.482872 13066 ts_tablet_manager.cc:1398] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.483062 13066 tablet_bootstrap.cc:492] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:20.484083 13066 tablet_bootstrap.cc:654] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.486013 13138 tablet_bootstrap.cc:492] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:20.487913 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:20.495373 9767 raft_consensus.cc:1261] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.497905 9504 raft_consensus.cc:1261] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.498335 7278 raft_consensus.cc:1203] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:20.498546 13135 tablet_bootstrap.cc:492] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:20.499161 12790 consensus_queue.cc:1035] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.499524 12785 consensus_queue.cc:1035] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.499687 13070 tablet_bootstrap.cc:492] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:20.499804 13070 ts_tablet_manager.cc:1367] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.002s
I20240503 07:59:20.500303 13070 raft_consensus.cc:348] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.500416 13070 raft_consensus.cc:374] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.500444 13070 raft_consensus.cc:729] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.500555 13070 consensus_queue.cc:260] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.500870 13070 ts_tablet_manager.cc:1398] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:20.501055 13070 tablet_bootstrap.cc:492] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:20.502071 13070 tablet_bootstrap.cc:654] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.505318 13066 tablet_bootstrap.cc:492] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:20.505426 13066 ts_tablet_manager.cc:1367] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 07:59:20.505955 13066 raft_consensus.cc:348] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.506080 13066 raft_consensus.cc:374] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.506112 13066 raft_consensus.cc:729] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.506232 13066 consensus_queue.cc:260] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.506559 13066 ts_tablet_manager.cc:1398] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.506655 13135 tablet_bootstrap.cc:654] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.506873 13066 tablet_bootstrap.cc:492] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:20.507930 9766 raft_consensus.cc:1203] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:20.508034 13066 tablet_bootstrap.cc:654] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.510562 6741 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 166a5a44983c43d581c0332878cebfdd (table rebalancer_dynamic_table [id=295d54d12ba2425d855dd1fa0513f8f7]). Expected version 1 got 0
I20240503 07:59:20.513929 6747 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 166a5a44983c43d581c0332878cebfdd (table rebalancer_dynamic_table [id=295d54d12ba2425d855dd1fa0513f8f7]). Expected version 1 got 0
I20240503 07:59:20.514498 9484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:20.515509 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:20.517165 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:20.520645 13066 tablet_bootstrap.cc:492] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:20.520679 13070 tablet_bootstrap.cc:492] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:20.520794 13070 ts_tablet_manager.cc:1367] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 07:59:20.520852 13066 ts_tablet_manager.cc:1367] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 07:59:20.521337 13070 raft_consensus.cc:348] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.521466 13070 raft_consensus.cc:374] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.521494 13070 raft_consensus.cc:729] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.521607 13070 consensus_queue.cc:260] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.521886 13070 ts_tablet_manager.cc:1398] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.528091 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:20.528483 9746 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:20.529271 13135 tablet_bootstrap.cc:492] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:20.529479 13135 ts_tablet_manager.cc:1367] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20240503 07:59:20.530042 13135 raft_consensus.cc:348] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.530509 13135 raft_consensus.cc:374] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.530609 13135 raft_consensus.cc:729] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.531059 13135 consensus_queue.cc:260] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.532361 13066 raft_consensus.cc:348] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.532480 13066 raft_consensus.cc:374] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.532508 13066 raft_consensus.cc:729] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.532627 13066 consensus_queue.cc:260] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.533064 13066 ts_tablet_manager.cc:1398] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240503 07:59:20.536579 13135 ts_tablet_manager.cc:1398] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:20.549813 9505 raft_consensus.cc:1203] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:20.560406 13138 tablet_bootstrap.cc:654] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.562286 13148 tablet.cc:1722] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:20.562763 13148 tablet.cc:1620] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:20.566704 7020 raft_consensus.cc:1203] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:20.570835 12578 tablet.cc:1722] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:20.571193 12578 tablet.cc:1620] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:20.572367 13151 tablet.cc:1722] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:20.573031 13151 tablet.cc:1620] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:20.573412 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=295d54d12ba2425d855dd1fa0513f8f7] alter complete (version 1)
I20240503 07:59:20.574872 6738 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 166a5a44983c43d581c0332878cebfdd (table rebalancer_dynamic_table [id=295d54d12ba2425d855dd1fa0513f8f7]). Expected version 1 got 0
I20240503 07:59:20.577314 13027 raft_consensus.cc:482] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:20.577330 9504 raft_consensus.cc:1203] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:20.577412 13027 raft_consensus.cc:504] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.577813 13027 leader_election.cc:288] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:20.578069 13138 tablet_bootstrap.cc:492] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:20.578179 13138 ts_tablet_manager.cc:1367] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.000s
I20240503 07:59:20.578459 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc7505d3ef6416c9d505bfe899f73ed" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:20.578591 9504 raft_consensus.cc:2454] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:20.578779 13138 raft_consensus.cc:348] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.578899 13138 raft_consensus.cc:374] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.578933 13138 raft_consensus.cc:729] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.578955 9703 leader_election.cc:302] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:20.579056 13138 consensus_queue.cc:260] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.579368 13138 ts_tablet_manager.cc:1398] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:20.579563 13138 tablet_bootstrap.cc:492] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:20.580000 13027 raft_consensus.cc:2781] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:20.580044 13027 raft_consensus.cc:482] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:20.580071 13027 raft_consensus.cc:3037] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.580560 13138 tablet_bootstrap.cc:654] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.580916 13027 raft_consensus.cc:504] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.581259 13027 leader_election.cc:288] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:20.582410 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc7505d3ef6416c9d505bfe899f73ed" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:20.582441 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc7505d3ef6416c9d505bfe899f73ed" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:20.582540 9357 raft_consensus.cc:2454] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:20.584995 13138 tablet_bootstrap.cc:492] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:20.585103 13138 ts_tablet_manager.cc:1367] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:20.585657 13138 raft_consensus.cc:348] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.585783 13138 raft_consensus.cc:374] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.585844 13138 raft_consensus.cc:729] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.586009 13138 consensus_queue.cc:260] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.587069 13138 ts_tablet_manager.cc:1398] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:20.587203 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc7505d3ef6416c9d505bfe899f73ed" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:20.587343 9504 raft_consensus.cc:3037] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.588327 9504 raft_consensus.cc:2454] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:20.591048 9703 leader_election.cc:302] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:20.591285 13027 raft_consensus.cc:2781] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.591349 13027 raft_consensus.cc:686] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:20.591462 13027 consensus_queue.cc:237] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:20.593400 6738 catalog_manager.cc:5484] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:20.612295 6888 consensus_queue.cc:237] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:20.613713 9767 raft_consensus.cc:1261] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:20.613765 7149 raft_consensus.cc:1261] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:20.614128 13087 consensus_queue.cc:1035] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:20.614300 12525 consensus_queue.cc:1035] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:20.616135 13172 raft_consensus.cc:2932] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:20.616576 7149 raft_consensus.cc:2932] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:20.616611 9767 raft_consensus.cc:2932] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:20.619728 6741 catalog_manager.cc:5484] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:20.623242 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b794508aa77492398e48d25edfb8aa3 with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:20.623591 13177 tablet_replica.cc:330] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:20.623776 13177 raft_consensus.cc:2229] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.623947 13177 raft_consensus.cc:2258] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.624478 13177 ts_tablet_manager.cc:1875] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.635236 9767 raft_consensus.cc:1203] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:20.635748 13177 ts_tablet_manager.cc:1888] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:20.635970 13177 log.cc:1196] T 5b794508aa77492398e48d25edfb8aa3 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5b794508aa77492398e48d25edfb8aa3
I20240503 07:59:20.636646 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 5b794508aa77492398e48d25edfb8aa3 (table rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]) successfully deleted
I20240503 07:59:20.687234 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:20.688040 12648 raft_consensus.cc:482] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:20.688159 12648 raft_consensus.cc:504] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.688588 12648 leader_election.cc:288] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:20.689163 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b607148b2a7b4b56b83e9606be49221b" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:20.689314 9634 raft_consensus.cc:2454] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:20.690686 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b607148b2a7b4b56b83e9606be49221b" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:20.690835 9357 raft_consensus.cc:2454] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:20.691507 7085 leader_election.cc:302] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:20.691699 12648 raft_consensus.cc:2781] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:20.691778 12648 raft_consensus.cc:482] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:20.691820 12648 raft_consensus.cc:3037] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.693033 12648 raft_consensus.cc:504] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.693415 12648 leader_election.cc:288] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:20.694175 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b607148b2a7b4b56b83e9606be49221b" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:20.694319 9634 raft_consensus.cc:3037] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.695475 9634 raft_consensus.cc:2454] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:20.696064 7084 leader_election.cc:302] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2; no voters:
I20240503 07:59:20.696596 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b607148b2a7b4b56b83e9606be49221b" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:20.696743 9357 raft_consensus.cc:3037] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.696993 12648 raft_consensus.cc:2781] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.697077 12648 raft_consensus.cc:686] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:20.697217 12648 consensus_queue.cc:237] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.697806 9357 raft_consensus.cc:2454] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:20.700290 6741 catalog_manager.cc:5484] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:20.700600 13148 tablet.cc:1722] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:20.700937 13148 tablet.cc:1620] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:20.702510 13151 tablet.cc:1722] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:20.702699 12578 tablet.cc:1722] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:20.702864 13151 tablet.cc:1620] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:20.702994 12578 tablet.cc:1620] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:20.708881 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=295d54d12ba2425d855dd1fa0513f8f7] alter complete (version 2)
I20240503 07:59:20.713296 9374 raft_consensus.cc:1203] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:20.810438 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:20.815938 13148 tablet.cc:1722] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:20.816344 12578 tablet.cc:1722] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:20.817272 13151 tablet.cc:1722] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:20.822628 6743 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=295d54d12ba2425d855dd1fa0513f8f7] alter complete (version 3)
I20240503 07:59:20.827437 12790 raft_consensus.cc:482] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:20.827579 12790 raft_consensus.cc:504] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.828061 12790 leader_election.cc:288] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:20.828578 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f55eb986bd4b7bbda10e683f3bfcad" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:20.828681 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f55eb986bd4b7bbda10e683f3bfcad" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:20.828752 9504 raft_consensus.cc:2454] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:20.828842 7149 raft_consensus.cc:2454] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:20.829062 9310 leader_election.cc:302] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:20.829303 12790 raft_consensus.cc:2781] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:20.829387 12790 raft_consensus.cc:482] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:20.829437 12790 raft_consensus.cc:3037] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.830296 12790 raft_consensus.cc:504] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.830693 12790 leader_election.cc:288] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:20.831025 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f55eb986bd4b7bbda10e683f3bfcad" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:20.831135 9505 raft_consensus.cc:3037] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.831106 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f55eb986bd4b7bbda10e683f3bfcad" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:20.831235 7149 raft_consensus.cc:3037] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.831986 9505 raft_consensus.cc:2454] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:20.832011 7149 raft_consensus.cc:2454] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:20.832309 9310 leader_election.cc:302] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:20.832476 12790 raft_consensus.cc:2781] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.832553 12790 raft_consensus.cc:686] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:20.832710 12790 consensus_queue.cc:237] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:20.835055 6743 catalog_manager.cc:5484] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:20.862785 9615 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:20.864184 9350 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:20.867049 9746 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:20.872356 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:20.879667 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:20.886157 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:20.890319 9484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:20.911450 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:20.924598 6743 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:20.924804 6743 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:20.927034 6743 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 166a5a44983c43d581c0332878cebfdd
I20240503 07:59:20.927789 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet 166a5a44983c43d581c0332878cebfdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:20 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:20.927914 9484 tablet_service.cc:1514] Processing DeleteTablet for tablet 166a5a44983c43d581c0332878cebfdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:20 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:20.928084 9746 tablet_service.cc:1514] Processing DeleteTablet for tablet 166a5a44983c43d581c0332878cebfdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:20 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:20.928452 13230 tablet_replica.cc:330] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:20.928654 13230 raft_consensus.cc:2229] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.928877 13230 raft_consensus.cc:2258] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.929576 13230 ts_tablet_manager.cc:1875] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:20.932247 13230 ts_tablet_manager.cc:1888] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:20.932433 13230 log.cc:1196] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/166a5a44983c43d581c0332878cebfdd
I20240503 07:59:20.932785 13230 ts_tablet_manager.cc:1909] T 166a5a44983c43d581c0332878cebfdd P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:20.933295 13231 tablet_replica.cc:330] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:20.933540 13231 raft_consensus.cc:2229] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.933634 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 166a5a44983c43d581c0332878cebfdd (table rebalancer_dynamic_table_new_name [id=295d54d12ba2425d855dd1fa0513f8f7]) successfully deleted
I20240503 07:59:20.933807 13231 raft_consensus.cc:2258] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.933904 13229 tablet_replica.cc:330] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:20.934062 13229 raft_consensus.cc:2229] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:20.934398 13229 raft_consensus.cc:2258] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.934551 13231 ts_tablet_manager.cc:1875] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:20.934926 13229 ts_tablet_manager.cc:1875] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:20.937036 13231 ts_tablet_manager.cc:1888] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:20.937145 13231 log.cc:1196] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/166a5a44983c43d581c0332878cebfdd
I20240503 07:59:20.937420 13231 ts_tablet_manager.cc:1909] T 166a5a44983c43d581c0332878cebfdd P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:20.938197 13229 ts_tablet_manager.cc:1888] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:20.938458 13229 log.cc:1196] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/166a5a44983c43d581c0332878cebfdd
I20240503 07:59:20.938900 13229 ts_tablet_manager.cc:1909] T 166a5a44983c43d581c0332878cebfdd P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:20.939667 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 166a5a44983c43d581c0332878cebfdd (table rebalancer_dynamic_table_new_name [id=295d54d12ba2425d855dd1fa0513f8f7]) successfully deleted
I20240503 07:59:20.939873 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 166a5a44983c43d581c0332878cebfdd (table rebalancer_dynamic_table_new_name [id=295d54d12ba2425d855dd1fa0513f8f7]) successfully deleted
I20240503 07:59:20.958918 7020 consensus_queue.cc:237] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:20.960382 7278 raft_consensus.cc:1261] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:20.960397 9634 raft_consensus.cc:1261] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:20.960968 11859 consensus_queue.cc:1035] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:20.961144 12707 consensus_queue.cc:1035] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:20.963119 13250 raft_consensus.cc:2932] T ea625a1834fb4a038a32a90d2fcbf1ac P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:20.963537 9634 raft_consensus.cc:2932] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:20.963634 7278 raft_consensus.cc:2932] T ea625a1834fb4a038a32a90d2fcbf1ac P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:20.966665 6743 catalog_manager.cc:5484] T ea625a1834fb4a038a32a90d2fcbf1ac P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:20.969961 7131 tablet_service.cc:1514] Processing DeleteTablet for tablet ea625a1834fb4a038a32a90d2fcbf1ac with delete_type TABLET_DATA_TOMBSTONED (TS 00a3f8e159c64327832d79fb32cdcca8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:20.970317 13256 tablet_replica.cc:330] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:20.970521 13256 raft_consensus.cc:2229] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.970682 13256 raft_consensus.cc:2258] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.971469 13256 ts_tablet_manager.cc:1875] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.973838 13256 ts_tablet_manager.cc:1888] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:20.973923 13256 log.cc:1196] T ea625a1834fb4a038a32a90d2fcbf1ac P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ea625a1834fb4a038a32a90d2fcbf1ac
I20240503 07:59:20.974498 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet ea625a1834fb4a038a32a90d2fcbf1ac (table rebalance_test_table_1 [id=7240d8c6aeff4067b4b95979e77f21da]) successfully deleted
I20240503 07:59:21.030303 6743 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:21.036024 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 784ca1c11559449999f4abc03c984017 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b4b57e27f9264a2a8864efe5d86a4ed8]), partition=
I20240503 07:59:21.036202 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 784ca1c11559449999f4abc03c984017 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b4b57e27f9264a2a8864efe5d86a4ed8]), partition=
I20240503 07:59:21.036360 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784ca1c11559449999f4abc03c984017. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.036484 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784ca1c11559449999f4abc03c984017. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.038466 13066 tablet_bootstrap.cc:492] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:21.039415 13066 tablet_bootstrap.cc:654] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.039788 13293 tablet_bootstrap.cc:492] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:21.040810 13293 tablet_bootstrap.cc:654] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.041361 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 784ca1c11559449999f4abc03c984017 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b4b57e27f9264a2a8864efe5d86a4ed8]), partition=
I20240503 07:59:21.041643 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784ca1c11559449999f4abc03c984017. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.042867 13066 tablet_bootstrap.cc:492] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:21.042968 13066 ts_tablet_manager.cc:1367] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:21.043454 13066 raft_consensus.cc:348] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:21.043569 13066 raft_consensus.cc:374] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.043610 13066 raft_consensus.cc:729] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.043741 13066 consensus_queue.cc:260] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:21.044031 13066 ts_tablet_manager.cc:1398] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.050068 13293 tablet_bootstrap.cc:492] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:21.050269 13293 ts_tablet_manager.cc:1367] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 07:59:21.050905 13293 raft_consensus.cc:348] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:21.052024 13293 raft_consensus.cc:374] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.052134 13293 raft_consensus.cc:729] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.052363 13293 consensus_queue.cc:260] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:21.053367 13293 ts_tablet_manager.cc:1398] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:21.059849 13296 tablet_bootstrap.cc:492] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:21.061040 13296 tablet_bootstrap.cc:654] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.062765 13296 tablet_bootstrap.cc:492] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:21.062880 13296 ts_tablet_manager.cc:1367] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:21.063406 13296 raft_consensus.cc:348] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:21.063532 13296 raft_consensus.cc:374] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.063562 13296 raft_consensus.cc:729] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.063701 13296 consensus_queue.cc:260] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:21.064004 13296 ts_tablet_manager.cc:1398] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:21.084656 9615 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:21.084683 9746 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:21.085019 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:21.087332 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:21.088174 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:21.091849 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:21.096448 12648 consensus_queue.cc:1035] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.098582 9350 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:21.102916 9357 raft_consensus.cc:3037] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.103045 12644 consensus_queue.cc:1035] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.105139 12507 consensus_queue.cc:1035] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.128430 12507 consensus_queue.cc:1035] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.133602 9484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:21.187788 7149 consensus_queue.cc:237] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:21.189328 6886 raft_consensus.cc:1261] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:21.190089 12648 consensus_queue.cc:1035] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:21.190793 9505 raft_consensus.cc:1261] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:21.191390 12648 consensus_queue.cc:1035] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:21.193311 12648 raft_consensus.cc:2932] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:21.193830 6886 raft_consensus.cc:2932] T c27273ae45c449018143cc55e65336a6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:21.195564 9505 raft_consensus.cc:2932] T c27273ae45c449018143cc55e65336a6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:21.197203 6739 catalog_manager.cc:5484] T c27273ae45c449018143cc55e65336a6 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:21.200433 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet c27273ae45c449018143cc55e65336a6 with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:21.201789 13336 tablet_replica.cc:330] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:21.201979 13336 raft_consensus.cc:2229] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:21.202145 13336 raft_consensus.cc:2258] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:21.202695 13336 ts_tablet_manager.cc:1875] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:21.207783 13336 ts_tablet_manager.cc:1888] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:21.207880 13336 log.cc:1196] T c27273ae45c449018143cc55e65336a6 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c27273ae45c449018143cc55e65336a6
I20240503 07:59:21.208629 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet c27273ae45c449018143cc55e65336a6 (table rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]) successfully deleted
I20240503 07:59:21.231462 13216 consensus_queue.cc:1035] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.234908 12790 consensus_queue.cc:1035] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.379405 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:21.386420 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 7009acabeee34602a3ce7462ed5a2d8a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7055be84cc0f4e368d98a9e128f121af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.386541 9353 tablet_service.cc:1467] Processing CreateTablet for tablet ad1a17fdefd3425aa0cda9a8fdc827c0 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7055be84cc0f4e368d98a9e128f121af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.386759 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7009acabeee34602a3ce7462ed5a2d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.386801 9614 tablet_service.cc:1467] Processing CreateTablet for tablet faee1f8d939e4487a014044772ba2daf (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7055be84cc0f4e368d98a9e128f121af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.386888 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1a17fdefd3425aa0cda9a8fdc827c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.386866 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 7009acabeee34602a3ce7462ed5a2d8a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7055be84cc0f4e368d98a9e128f121af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.387053 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faee1f8d939e4487a014044772ba2daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.387976 9745 tablet_service.cc:1467] Processing CreateTablet for tablet ad1a17fdefd3425aa0cda9a8fdc827c0 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7055be84cc0f4e368d98a9e128f121af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.388242 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1a17fdefd3425aa0cda9a8fdc827c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.389349 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 7009acabeee34602a3ce7462ed5a2d8a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7055be84cc0f4e368d98a9e128f121af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.389575 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7009acabeee34602a3ce7462ed5a2d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.390444 13293 tablet_bootstrap.cc:492] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:21.391670 13066 tablet_bootstrap.cc:492] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:21.391917 13293 tablet_bootstrap.cc:654] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.392578 13066 tablet_bootstrap.cc:654] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.393193 9484 tablet_service.cc:1467] Processing CreateTablet for tablet faee1f8d939e4487a014044772ba2daf (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7055be84cc0f4e368d98a9e128f121af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.393330 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7009acabeee34602a3ce7462ed5a2d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.393477 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faee1f8d939e4487a014044772ba2daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.394289 6999 tablet_service.cc:1467] Processing CreateTablet for tablet ad1a17fdefd3425aa0cda9a8fdc827c0 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7055be84cc0f4e368d98a9e128f121af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.394573 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1a17fdefd3425aa0cda9a8fdc827c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.394949 13380 tablet_bootstrap.cc:492] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:21.397135 13380 tablet_bootstrap.cc:654] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.397195 13293 tablet_bootstrap.cc:492] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:21.397279 13293 ts_tablet_manager.cc:1367] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:21.397770 13384 tablet_bootstrap.cc:492] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:21.397733 13293 raft_consensus.cc:348] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:21.397842 13293 raft_consensus.cc:374] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.397876 13293 raft_consensus.cc:729] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.397974 13293 consensus_queue.cc:260] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:21.398236 13293 ts_tablet_manager.cc:1398] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.398418 13293 tablet_bootstrap.cc:492] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:21.399009 13384 tablet_bootstrap.cc:654] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.399248 13293 tablet_bootstrap.cc:654] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.399425 9350 tablet_service.cc:1467] Processing CreateTablet for tablet faee1f8d939e4487a014044772ba2daf (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7055be84cc0f4e368d98a9e128f121af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.399680 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faee1f8d939e4487a014044772ba2daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.399820 13383 tablet_bootstrap.cc:492] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:21.400264 13384 tablet_bootstrap.cc:492] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:21.400346 13384 ts_tablet_manager.cc:1367] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:21.400758 13383 tablet_bootstrap.cc:654] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.400787 13384 raft_consensus.cc:348] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:21.400888 13384 raft_consensus.cc:374] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.400921 13384 raft_consensus.cc:729] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.401044 13384 consensus_queue.cc:260] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:21.401345 13384 ts_tablet_manager.cc:1398] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.402508 13380 tablet_bootstrap.cc:492] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:21.402508 13383 tablet_bootstrap.cc:492] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:21.402604 13383 ts_tablet_manager.cc:1367] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:21.402604 13380 ts_tablet_manager.cc:1367] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:21.403086 13380 raft_consensus.cc:348] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.403220 13380 raft_consensus.cc:374] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.403260 13380 raft_consensus.cc:729] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.403229 13383 raft_consensus.cc:348] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.403322 13383 raft_consensus.cc:374] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.403398 13383 raft_consensus.cc:729] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.403380 13380 consensus_queue.cc:260] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.403550 13383 consensus_queue.cc:260] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.403662 13380 ts_tablet_manager.cc:1398] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:21.403822 13380 tablet_bootstrap.cc:492] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:21.403968 13383 ts_tablet_manager.cc:1398] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.404170 13293 tablet_bootstrap.cc:492] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:21.404249 13293 ts_tablet_manager.cc:1367] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:21.404628 13380 tablet_bootstrap.cc:654] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.404654 13293 raft_consensus.cc:348] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.404760 13293 raft_consensus.cc:374] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.404791 13293 raft_consensus.cc:729] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.404886 13293 consensus_queue.cc:260] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.405154 13293 ts_tablet_manager.cc:1398] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.406097 13066 tablet_bootstrap.cc:492] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:21.406186 13066 ts_tablet_manager.cc:1367] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:21.406688 13066 raft_consensus.cc:348] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.406795 13066 raft_consensus.cc:374] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.406823 13066 raft_consensus.cc:729] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.406946 13066 consensus_queue.cc:260] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.407215 13066 ts_tablet_manager.cc:1398] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.407402 13066 tablet_bootstrap.cc:492] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:21.408241 13314 raft_consensus.cc:482] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:21.408316 13066 tablet_bootstrap.cc:654] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.408340 13314 raft_consensus.cc:504] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:21.408792 13314 leader_election.cc:288] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:21.409587 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784ca1c11559449999f4abc03c984017" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:21.409718 9374 raft_consensus.cc:2454] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:21.409976 9701 leader_election.cc:302] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:21.410516 13315 raft_consensus.cc:2781] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:21.410569 13315 raft_consensus.cc:482] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:21.410605 13315 raft_consensus.cc:3037] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.411761 13315 raft_consensus.cc:504] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:21.412117 13315 leader_election.cc:288] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:21.412680 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784ca1c11559449999f4abc03c984017" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:21.412806 9374 raft_consensus.cc:3037] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.413874 9374 raft_consensus.cc:2454] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:21.414446 9701 leader_election.cc:302] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:21.414808 13315 raft_consensus.cc:2781] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:21.414885 13315 raft_consensus.cc:686] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:21.415027 13066 tablet_bootstrap.cc:492] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:21.415028 13315 consensus_queue.cc:237] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:21.415110 13066 ts_tablet_manager.cc:1367] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:21.415449 13380 tablet_bootstrap.cc:492] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:21.415552 13380 ts_tablet_manager.cc:1367] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240503 07:59:21.415594 13066 raft_consensus.cc:348] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:21.415693 13066 raft_consensus.cc:374] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.415722 13066 raft_consensus.cc:729] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.415838 13066 consensus_queue.cc:260] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:21.415872 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784ca1c11559449999f4abc03c984017" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:21.415877 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784ca1c11559449999f4abc03c984017" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:21.416029 7282 raft_consensus.cc:3037] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.416131 13066 ts_tablet_manager.cc:1398] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.416303 13066 tablet_bootstrap.cc:492] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:21.417063 7282 raft_consensus.cc:2454] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:21.417856 13066 tablet_bootstrap.cc:654] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.418668 13380 raft_consensus.cc:348] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.418888 13380 raft_consensus.cc:374] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.419090 13380 raft_consensus.cc:729] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.419272 13380 consensus_queue.cc:260] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.422183 13380 ts_tablet_manager.cc:1398] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 07:59:21.423717 13066 tablet_bootstrap.cc:492] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:21.423846 13066 ts_tablet_manager.cc:1367] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 07:59:21.424273 6747 catalog_manager.cc:5484] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:21.424399 13066 raft_consensus.cc:348] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.424517 13066 raft_consensus.cc:374] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.424543 13066 raft_consensus.cc:729] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.424670 13066 consensus_queue.cc:260] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.424935 13066 ts_tablet_manager.cc:1398] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:21.452867 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:21.459935 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:21.462821 9615 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:21.468134 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:21.471004 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:21.471832 9350 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:21.479362 9745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:21.498693 9484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:21.510609 12934 raft_consensus.cc:482] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:21.510752 12934 raft_consensus.cc:504] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.511225 12934 leader_election.cc:288] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:21.511533 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faee1f8d939e4487a014044772ba2daf" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:21.511688 9505 raft_consensus.cc:2454] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:21.511705 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faee1f8d939e4487a014044772ba2daf" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:21.511863 9374 raft_consensus.cc:2454] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:21.511994 9572 leader_election.cc:302] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:21.512894 12934 raft_consensus.cc:2781] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:21.512962 12934 raft_consensus.cc:482] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:21.512995 12934 raft_consensus.cc:3037] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.514245 12934 raft_consensus.cc:504] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.514638 12934 leader_election.cc:288] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:21.514942 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faee1f8d939e4487a014044772ba2daf" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:21.515060 9505 raft_consensus.cc:3037] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.515365 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faee1f8d939e4487a014044772ba2daf" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:21.515480 9374 raft_consensus.cc:3037] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.516093 9505 raft_consensus.cc:2454] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:21.516474 9374 raft_consensus.cc:2454] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:21.516556 9572 leader_election.cc:302] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:21.518703 12934 raft_consensus.cc:2781] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:21.518913 12934 raft_consensus.cc:686] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:21.519053 12934 consensus_queue.cc:237] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.521281 6747 catalog_manager.cc:5484] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:21.545882 13314 raft_consensus.cc:482] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:21.546028 13314 raft_consensus.cc:504] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.546492 13314 leader_election.cc:288] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:21.546797 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7009acabeee34602a3ce7462ed5a2d8a" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:21.546934 9636 raft_consensus.cc:2454] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:21.546988 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7009acabeee34602a3ce7462ed5a2d8a" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:21.547123 9374 raft_consensus.cc:2454] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:21.547298 9700 leader_election.cc:302] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:21.547446 13314 raft_consensus.cc:2781] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:21.547500 13314 raft_consensus.cc:482] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:21.547536 13314 raft_consensus.cc:3037] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.548750 13314 raft_consensus.cc:504] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.549130 13314 leader_election.cc:288] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:21.549362 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7009acabeee34602a3ce7462ed5a2d8a" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:21.549484 9636 raft_consensus.cc:3037] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.549702 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7009acabeee34602a3ce7462ed5a2d8a" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:21.549816 9374 raft_consensus.cc:3037] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.550557 9636 raft_consensus.cc:2454] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:21.550959 9374 raft_consensus.cc:2454] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:21.550953 9700 leader_election.cc:302] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:21.551400 13314 raft_consensus.cc:2781] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:21.551491 13314 raft_consensus.cc:686] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:21.551627 13314 consensus_queue.cc:237] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:21.553615 13216 raft_consensus.cc:482] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:21.553721 13216 raft_consensus.cc:504] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:21.554152 13216 leader_election.cc:288] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:21.554446 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1a17fdefd3425aa0cda9a8fdc827c0" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:21.554292 6747 catalog_manager.cc:5484] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:21.554553 9767 raft_consensus.cc:2454] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:21.554692 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1a17fdefd3425aa0cda9a8fdc827c0" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:21.554836 7020 raft_consensus.cc:2454] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:21.555063 9310 leader_election.cc:302] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:21.555227 13216 raft_consensus.cc:2781] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:21.555285 13216 raft_consensus.cc:482] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:21.555318 13216 raft_consensus.cc:3037] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.556311 13216 raft_consensus.cc:504] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:21.556677 13216 leader_election.cc:288] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:21.557001 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1a17fdefd3425aa0cda9a8fdc827c0" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:21.557096 9767 raft_consensus.cc:3037] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.557690 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1a17fdefd3425aa0cda9a8fdc827c0" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:21.557816 7020 raft_consensus.cc:3037] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.558022 9767 raft_consensus.cc:2454] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:21.558872 9310 leader_election.cc:302] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:21.559006 7020 raft_consensus.cc:2454] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:21.559702 13216 raft_consensus.cc:2781] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:21.559813 13216 raft_consensus.cc:686] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:21.559952 13216 consensus_queue.cc:237] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:21.561930 6743 catalog_manager.cc:5484] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:21.567312 7149 consensus_queue.cc:237] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:21.568542 9767 raft_consensus.cc:1261] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:21.568574 7020 raft_consensus.cc:1261] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:21.568926 13335 consensus_queue.cc:1035] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:21.569118 12644 consensus_queue.cc:1035] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:21.570951 13335 raft_consensus.cc:2932] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:21.571365 7020 raft_consensus.cc:2932] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:21.571604 9767 raft_consensus.cc:2932] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:21.573966 6743 catalog_manager.cc:5484] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:21.577062 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2386cd009847d9a1fe7a959f47086c with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:21.577620 13429 tablet_replica.cc:330] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:21.577783 13429 raft_consensus.cc:2229] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:21.577948 13429 raft_consensus.cc:2258] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:21.578627 13429 ts_tablet_manager.cc:1875] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:21.581194 13429 ts_tablet_manager.cc:1888] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:21.581279 13429 log.cc:1196] T 7c2386cd009847d9a1fe7a959f47086c P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7c2386cd009847d9a1fe7a959f47086c
I20240503 07:59:21.581838 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 7c2386cd009847d9a1fe7a959f47086c (table rebalance_test_table_9 [id=3e242002464141aaa2f77f2115a129ff]) successfully deleted
I20240503 07:59:21.612300 6747 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:21.616991 7282 raft_consensus.cc:1261] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:21.617004 9374 raft_consensus.cc:1261] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:21.617455 13315 consensus_queue.cc:1035] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.617772 13315 consensus_queue.cc:1035] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.622211 6739 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet 784ca1c11559449999f4abc03c984017 (table rebalancer_dynamic_table [id=b4b57e27f9264a2a8864efe5d86a4ed8]). Expected version 1 got 0
I20240503 07:59:21.622649 6747 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet 784ca1c11559449999f4abc03c984017 (table rebalancer_dynamic_table [id=b4b57e27f9264a2a8864efe5d86a4ed8]). Expected version 1 got 0
I20240503 07:59:21.624228 13455 tablet.cc:1722] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:21.624544 13453 tablet.cc:1722] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:21.625000 13453 tablet.cc:1620] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:21.625330 13455 tablet.cc:1620] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:21.625661 6741 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet 784ca1c11559449999f4abc03c984017 (table rebalancer_dynamic_table [id=b4b57e27f9264a2a8864efe5d86a4ed8]). Expected version 1 got 0
I20240503 07:59:21.627771 6741 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet 784ca1c11559449999f4abc03c984017 (table rebalancer_dynamic_table [id=b4b57e27f9264a2a8864efe5d86a4ed8]). Expected version 1 got 0
I20240503 07:59:21.631050 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=b4b57e27f9264a2a8864efe5d86a4ed8] alter complete (version 1)
I20240503 07:59:21.634845 13456 tablet.cc:1722] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:21.635687 13456 tablet.cc:1620] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:21.688295 9350 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:21.689628 9615 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:21.696888 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:21.698199 9745 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:21.698489 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:21.701495 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:21.713825 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:21.733337 9484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:21.736207 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:21.747206 13453 tablet.cc:1722] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.747612 13453 tablet.cc:1620] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:21.747718 13455 tablet.cc:1722] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.748024 13455 tablet.cc:1620] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:21.751169 13456 tablet.cc:1722] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.751474 13456 tablet.cc:1620] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:21.754022 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=b4b57e27f9264a2a8864efe5d86a4ed8] alter complete (version 2)
I20240503 07:59:21.794606 7149 consensus_queue.cc:237] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:21.795938 7020 raft_consensus.cc:1261] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:21.796310 9636 raft_consensus.cc:1261] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:21.796382 12644 consensus_queue.cc:1035] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:21.796711 13335 consensus_queue.cc:1035] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:21.798323 12644 raft_consensus.cc:2932] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:21.798753 7020 raft_consensus.cc:2932] T 5a838dc6e3114be5aceac51d567bc645 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:21.799016 9636 raft_consensus.cc:2932] T 5a838dc6e3114be5aceac51d567bc645 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:21.801651 6739 catalog_manager.cc:5484] T 5a838dc6e3114be5aceac51d567bc645 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:21.804868 9484 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a838dc6e3114be5aceac51d567bc645 with delete_type TABLET_DATA_TOMBSTONED (TS cb73994515af4880b05383e78368bb4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:21.805392 13511 tablet_replica.cc:330] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:21.805605 13511 raft_consensus.cc:2229] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:21.805773 13511 raft_consensus.cc:2258] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:21.806279 13511 ts_tablet_manager.cc:1875] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:21.809897 13511 ts_tablet_manager.cc:1888] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:21.809980 13511 log.cc:1196] T 5a838dc6e3114be5aceac51d567bc645 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/5a838dc6e3114be5aceac51d567bc645
I20240503 07:59:21.810505 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 5a838dc6e3114be5aceac51d567bc645 (table rebalancer_extra_table_12 [id=4b3fe402f6344742a8841ffd09a9f966]) successfully deleted
I20240503 07:59:21.859899 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:21.867118 13453 tablet.cc:1722] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.869899 13455 tablet.cc:1722] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.872849 13456 tablet.cc:1722] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.874985 6742 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=b4b57e27f9264a2a8864efe5d86a4ed8] alter complete (version 3)
I20240503 07:59:21.900589 9350 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:21.903818 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:21.909524 9745 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:21.910143 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:21.912708 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:21.922828 9615 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:21.935317 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:21.936793 12790 consensus_queue.cc:1035] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.946941 9484 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:21.949084 13216 consensus_queue.cc:1035] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.975535 13314 consensus_queue.cc:1035] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.980036 13314 consensus_queue.cc:1035] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.981093 6742 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:21.981228 6742 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:21.983690 6742 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 784ca1c11559449999f4abc03c984017
I20240503 07:59:21.984514 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet 784ca1c11559449999f4abc03c984017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:21 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:21.985000 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 784ca1c11559449999f4abc03c984017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:21 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:21.985186 13429 tablet_replica.cc:330] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:21.985175 9745 tablet_service.cc:1514] Processing DeleteTablet for tablet 784ca1c11559449999f4abc03c984017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:21 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:21.985342 13429 raft_consensus.cc:2229] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:21.985467 13429 raft_consensus.cc:2258] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:21.985487 13569 tablet_replica.cc:330] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:21.985615 13569 raft_consensus.cc:2229] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:21.985792 13569 raft_consensus.cc:2258] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:21.986308 13569 ts_tablet_manager.cc:1875] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:21.986383 13429 ts_tablet_manager.cc:1875] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:21.986588 13568 tablet_replica.cc:330] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:21.986730 13568 raft_consensus.cc:2229] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:21.986838 13568 raft_consensus.cc:2258] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:21.987396 13568 ts_tablet_manager.cc:1875] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:21.988655 13429 ts_tablet_manager.cc:1888] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:21.988660 13569 ts_tablet_manager.cc:1888] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:21.988721 13429 log.cc:1196] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/784ca1c11559449999f4abc03c984017
I20240503 07:59:21.988725 13569 log.cc:1196] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/784ca1c11559449999f4abc03c984017
I20240503 07:59:21.988970 13569 ts_tablet_manager.cc:1909] T 784ca1c11559449999f4abc03c984017 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:21.988968 13429 ts_tablet_manager.cc:1909] T 784ca1c11559449999f4abc03c984017 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:21.989771 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 784ca1c11559449999f4abc03c984017 (table rebalancer_dynamic_table_new_name [id=b4b57e27f9264a2a8864efe5d86a4ed8]) successfully deleted
I20240503 07:59:21.989962 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 784ca1c11559449999f4abc03c984017 (table rebalancer_dynamic_table_new_name [id=b4b57e27f9264a2a8864efe5d86a4ed8]) successfully deleted
I20240503 07:59:21.990180 13568 ts_tablet_manager.cc:1888] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:21.990265 13568 log.cc:1196] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/784ca1c11559449999f4abc03c984017
I20240503 07:59:21.990543 13568 ts_tablet_manager.cc:1909] T 784ca1c11559449999f4abc03c984017 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:21.991236 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 784ca1c11559449999f4abc03c984017 (table rebalancer_dynamic_table_new_name [id=b4b57e27f9264a2a8864efe5d86a4ed8]) successfully deleted
I20240503 07:59:22.003656 13406 consensus_queue.cc:1035] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.006800 13406 consensus_queue.cc:1035] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.012781 7019 consensus_queue.cc:237] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:22.014101 9767 raft_consensus.cc:1261] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:22.014226 7282 raft_consensus.cc:1261] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:22.014626 13563 consensus_queue.cc:1035] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:22.014837 11859 consensus_queue.cc:1035] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:22.016669 13563 raft_consensus.cc:2932] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:22.016999 9767 raft_consensus.cc:2932] T 7a2312d0a7934056bf438a78e6641226 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:22.018532 7282 raft_consensus.cc:2932] T 7a2312d0a7934056bf438a78e6641226 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:22.020243 6742 catalog_manager.cc:5484] T 7a2312d0a7934056bf438a78e6641226 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:22.025077 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2312d0a7934056bf438a78e6641226 with delete_type TABLET_DATA_TOMBSTONED (TS 159177b3caeb434baa98e04136d050b2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:22.025494 13597 tablet_replica.cc:330] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:22.025698 13597 raft_consensus.cc:2229] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:22.025866 13597 raft_consensus.cc:2258] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.026427 13597 ts_tablet_manager.cc:1875] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:22.028847 13597 ts_tablet_manager.cc:1888] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:22.028929 13597 log.cc:1196] T 7a2312d0a7934056bf438a78e6641226 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/7a2312d0a7934056bf438a78e6641226
I20240503 07:59:22.029455 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 7a2312d0a7934056bf438a78e6641226 (table rebalancer_extra_table_6 [id=476bb7319d81414d9f735392a591b6a6]) successfully deleted
I20240503 07:59:22.078698 6745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:22.085445 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 730eb09f729844ce9f5952c46031c0d8 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=179f1c255c4d4e009e7f8119bc4e18be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.085661 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 730eb09f729844ce9f5952c46031c0d8 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=179f1c255c4d4e009e7f8119bc4e18be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.085788 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730eb09f729844ce9f5952c46031c0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.085914 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730eb09f729844ce9f5952c46031c0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.085927 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 287f2a8f05164e0eb722d53242e35023 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=179f1c255c4d4e009e7f8119bc4e18be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.086237 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287f2a8f05164e0eb722d53242e35023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.088393 13634 tablet_bootstrap.cc:492] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:22.088680 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 287f2a8f05164e0eb722d53242e35023 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=179f1c255c4d4e009e7f8119bc4e18be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.088685 9484 tablet_service.cc:1467] Processing CreateTablet for tablet 730eb09f729844ce9f5952c46031c0d8 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=179f1c255c4d4e009e7f8119bc4e18be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.088793 13635 tablet_bootstrap.cc:492] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:22.088929 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287f2a8f05164e0eb722d53242e35023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.088954 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730eb09f729844ce9f5952c46031c0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.089285 9483 tablet_service.cc:1467] Processing CreateTablet for tablet fb1bff0a8228454f95c14762536a4c82 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=179f1c255c4d4e009e7f8119bc4e18be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.089537 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1bff0a8228454f95c14762536a4c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.090003 13634 tablet_bootstrap.cc:654] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.091233 13635 tablet_bootstrap.cc:654] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.091454 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 287f2a8f05164e0eb722d53242e35023 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=179f1c255c4d4e009e7f8119bc4e18be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.091733 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287f2a8f05164e0eb722d53242e35023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.092864 13637 tablet_bootstrap.cc:492] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:22.093932 7260 tablet_service.cc:1467] Processing CreateTablet for tablet fb1bff0a8228454f95c14762536a4c82 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=179f1c255c4d4e009e7f8119bc4e18be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.094261 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1bff0a8228454f95c14762536a4c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.094266 13637 tablet_bootstrap.cc:654] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.095404 13634 tablet_bootstrap.cc:492] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:22.095562 13634 ts_tablet_manager.cc:1367] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:22.096108 6999 tablet_service.cc:1467] Processing CreateTablet for tablet fb1bff0a8228454f95c14762536a4c82 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=179f1c255c4d4e009e7f8119bc4e18be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.096098 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:22.096400 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1bff0a8228454f95c14762536a4c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.101094 13634 raft_consensus.cc:348] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.101367 13634 raft_consensus.cc:374] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.101462 13634 raft_consensus.cc:729] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.101711 13634 consensus_queue.cc:260] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.102512 13634 ts_tablet_manager.cc:1398] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:22.103561 13635 tablet_bootstrap.cc:492] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:22.109866 13635 ts_tablet_manager.cc:1367] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20240503 07:59:22.110167 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 75d347ec0591437bbeb226a6b3096611 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75456d8f3f454482a616b92801cc178d]), partition=
I20240503 07:59:22.110215 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 75d347ec0591437bbeb226a6b3096611 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75456d8f3f454482a616b92801cc178d]), partition=
I20240503 07:59:22.110532 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d347ec0591437bbeb226a6b3096611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.110574 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d347ec0591437bbeb226a6b3096611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.110607 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 75d347ec0591437bbeb226a6b3096611 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75456d8f3f454482a616b92801cc178d]), partition=
I20240503 07:59:22.110556 13635 raft_consensus.cc:348] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.110673 13635 raft_consensus.cc:374] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.110705 13635 raft_consensus.cc:729] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.110827 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d347ec0591437bbeb226a6b3096611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.110833 13635 consensus_queue.cc:260] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.111264 13635 ts_tablet_manager.cc:1398] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:22.111440 13635 tablet_bootstrap.cc:492] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:22.112461 13635 tablet_bootstrap.cc:654] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.113101 13634 tablet_bootstrap.cc:492] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:22.114260 13634 tablet_bootstrap.cc:654] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.114668 13641 tablet_bootstrap.cc:492] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:22.117257 13634 tablet_bootstrap.cc:492] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:22.117353 13634 ts_tablet_manager.cc:1367] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:22.117861 13641 tablet_bootstrap.cc:654] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.117810 13634 raft_consensus.cc:348] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:22.117923 13634 raft_consensus.cc:374] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.117952 13634 raft_consensus.cc:729] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.118077 13634 consensus_queue.cc:260] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:22.117851 13642 tablet_bootstrap.cc:492] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:22.118378 13634 ts_tablet_manager.cc:1398] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:22.119313 13635 tablet_bootstrap.cc:492] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:22.119403 13635 ts_tablet_manager.cc:1367] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:22.119843 13635 raft_consensus.cc:348] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.119968 13635 raft_consensus.cc:374] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.119999 13635 raft_consensus.cc:729] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.120105 13635 consensus_queue.cc:260] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.120400 13635 ts_tablet_manager.cc:1398] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:22.120872 13637 tablet_bootstrap.cc:492] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:22.120960 13637 ts_tablet_manager.cc:1367] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240503 07:59:22.121428 13637 raft_consensus.cc:348] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.121533 13637 raft_consensus.cc:374] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.121562 13637 raft_consensus.cc:729] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.122098 13637 consensus_queue.cc:260] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.122435 13637 ts_tablet_manager.cc:1398] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:22.123001 13637 tablet_bootstrap.cc:492] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:22.123100 13642 tablet_bootstrap.cc:654] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.123174 13641 tablet_bootstrap.cc:492] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:22.123277 13641 ts_tablet_manager.cc:1367] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:22.123967 13637 tablet_bootstrap.cc:654] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.123792 13641 raft_consensus.cc:348] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:22.124074 13641 raft_consensus.cc:374] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.124117 13641 raft_consensus.cc:729] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.124248 13641 consensus_queue.cc:260] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:22.124547 13641 ts_tablet_manager.cc:1398] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:22.125506 13636 tablet_bootstrap.cc:492] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:22.128157 13636 tablet_bootstrap.cc:654] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.128510 13642 tablet_bootstrap.cc:492] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:22.128670 13642 ts_tablet_manager.cc:1367] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:22.129405 13642 raft_consensus.cc:348] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:22.129622 13642 raft_consensus.cc:374] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.129868 13642 raft_consensus.cc:729] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.130232 13642 consensus_queue.cc:260] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:22.130901 13642 ts_tablet_manager.cc:1398] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:22.131048 13636 tablet_bootstrap.cc:492] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:22.131160 13636 ts_tablet_manager.cc:1367] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:22.131727 13636 raft_consensus.cc:348] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.131891 13636 raft_consensus.cc:374] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.131990 13636 raft_consensus.cc:729] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.132187 13636 consensus_queue.cc:260] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.132541 13636 ts_tablet_manager.cc:1398] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:22.132830 13636 tablet_bootstrap.cc:492] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:22.133971 13636 tablet_bootstrap.cc:654] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.135468 13637 tablet_bootstrap.cc:492] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:22.135627 13637 ts_tablet_manager.cc:1367] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:22.136230 13637 raft_consensus.cc:348] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:22.136454 13637 raft_consensus.cc:374] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.136538 13637 raft_consensus.cc:729] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.136737 13637 consensus_queue.cc:260] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:22.137193 13637 ts_tablet_manager.cc:1398] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:22.137538 13637 tablet_bootstrap.cc:492] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:22.137969 13636 tablet_bootstrap.cc:492] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:22.138056 13636 ts_tablet_manager.cc:1367] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:22.138672 13637 tablet_bootstrap.cc:654] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.138634 13636 raft_consensus.cc:348] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:22.138741 13636 raft_consensus.cc:374] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.138774 13636 raft_consensus.cc:729] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.138890 13636 consensus_queue.cc:260] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:22.139376 13636 ts_tablet_manager.cc:1398] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:22.139585 13636 tablet_bootstrap.cc:492] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:22.141940 13636 tablet_bootstrap.cc:654] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.142007 13637 tablet_bootstrap.cc:492] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:22.142107 13637 ts_tablet_manager.cc:1367] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:22.142836 13637 raft_consensus.cc:348] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.143055 13637 raft_consensus.cc:374] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.143138 13637 raft_consensus.cc:729] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.143321 13637 consensus_queue.cc:260] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.143740 13637 ts_tablet_manager.cc:1398] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:22.144330 13636 tablet_bootstrap.cc:492] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:22.144470 13636 ts_tablet_manager.cc:1367] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 07:59:22.145057 13636 raft_consensus.cc:348] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:22.145176 13636 raft_consensus.cc:374] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.145212 13636 raft_consensus.cc:729] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.145373 13636 consensus_queue.cc:260] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:22.145797 13636 ts_tablet_manager.cc:1398] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:22.161890 9615 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:22.170430 6999 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:22.170956 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:22.180490 9350 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:22.186916 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:22.189406 9485 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:22.191028 13418 raft_consensus.cc:482] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.191135 13418 raft_consensus.cc:504] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.191577 13418 leader_election.cc:288] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:22.192073 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287f2a8f05164e0eb722d53242e35023" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:22.192221 7282 raft_consensus.cc:2454] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:22.192320 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287f2a8f05164e0eb722d53242e35023" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:22.192463 9767 raft_consensus.cc:2454] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:22.192482 9441 leader_election.cc:302] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:22.193186 13418 raft_consensus.cc:2781] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.193245 13418 raft_consensus.cc:482] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.193277 13418 raft_consensus.cc:3037] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.194476 13418 raft_consensus.cc:504] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.194845 13418 leader_election.cc:288] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:22.195362 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287f2a8f05164e0eb722d53242e35023" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:22.195492 9767 raft_consensus.cc:3037] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.195559 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:22.196528 9767 raft_consensus.cc:2454] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:22.196877 9441 leader_election.cc:302] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:22.197351 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287f2a8f05164e0eb722d53242e35023" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:22.197489 7282 raft_consensus.cc:3037] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.199457 13418 raft_consensus.cc:2781] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.199549 13418 raft_consensus.cc:686] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:22.199698 13418 consensus_queue.cc:237] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.200798 7282 raft_consensus.cc:2454] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:22.206815 6746 catalog_manager.cc:5484] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:22.222921 9746 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:22.236393 13591 raft_consensus.cc:482] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.236542 13591 raft_consensus.cc:504] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:22.237004 13591 leader_election.cc:288] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:22.243265 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1bff0a8228454f95c14762536a4c82" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:22.243432 9505 raft_consensus.cc:2454] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:22.243911 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1bff0a8228454f95c14762536a4c82" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:22.244042 7282 raft_consensus.cc:2454] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:22.245187 6956 leader_election.cc:302] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:22.245504 13591 raft_consensus.cc:2781] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.245563 13591 raft_consensus.cc:482] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.245597 13591 raft_consensus.cc:3037] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.246815 13591 raft_consensus.cc:504] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:22.247179 13591 leader_election.cc:288] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:22.247624 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1bff0a8228454f95c14762536a4c82" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:22.247752 9505 raft_consensus.cc:3037] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.247815 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1bff0a8228454f95c14762536a4c82" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:22.247936 7282 raft_consensus.cc:3037] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.248700 9505 raft_consensus.cc:2454] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:22.248742 7282 raft_consensus.cc:2454] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:22.249045 6956 leader_election.cc:302] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:22.249681 13591 raft_consensus.cc:2781] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.249807 13591 raft_consensus.cc:686] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:22.249958 13591 consensus_queue.cc:237] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:22.252790 6746 catalog_manager.cc:5484] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:22.268157 13406 raft_consensus.cc:482] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.268313 13406 raft_consensus.cc:504] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:22.268739 13406 leader_election.cc:288] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:22.269104 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d347ec0591437bbeb226a6b3096611" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:22.269115 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d347ec0591437bbeb226a6b3096611" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:22.269258 9357 raft_consensus.cc:2454] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:22.269282 7282 raft_consensus.cc:2454] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:22.269579 9570 leader_election.cc:302] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:22.269793 13406 raft_consensus.cc:2781] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.269865 13406 raft_consensus.cc:482] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.269907 13406 raft_consensus.cc:3037] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.271055 13406 raft_consensus.cc:504] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:22.271446 13406 leader_election.cc:288] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:22.271875 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d347ec0591437bbeb226a6b3096611" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:22.271997 7282 raft_consensus.cc:3037] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.273130 7282 raft_consensus.cc:2454] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:22.273401 9572 leader_election.cc:302] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:22.273571 13406 raft_consensus.cc:2781] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.273638 13406 raft_consensus.cc:686] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:22.273597 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d347ec0591437bbeb226a6b3096611" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:22.273710 9357 raft_consensus.cc:3037] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.273775 13406 consensus_queue.cc:237] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:22.275588 9357 raft_consensus.cc:2454] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:22.276242 6746 catalog_manager.cc:5484] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:22.290674 9767 consensus_queue.cc:237] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:22.292021 9357 raft_consensus.cc:1261] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:22.292135 7149 raft_consensus.cc:1261] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:22.292538 13315 consensus_queue.cc:1035] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:22.292723 13314 consensus_queue.cc:1035] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:22.294757 13314 raft_consensus.cc:2932] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:22.296824 7149 raft_consensus.cc:2932] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:22.297674 9357 raft_consensus.cc:2932] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:22.298019 6746 catalog_manager.cc:5484] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index 2 to 3, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:22.301774 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7d7f7ee4384a2ba8bd0dac0ddad757 with delete_type TABLET_DATA_TOMBSTONED (TS c4218904dd7e42d2be95b901a6e5db92 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:22.302595 13689 tablet_replica.cc:330] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:22.302758 13689 raft_consensus.cc:2229] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:22.302883 13689 raft_consensus.cc:2258] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.303336 13689 ts_tablet_manager.cc:1875] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:22.305816 13689 ts_tablet_manager.cc:1888] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:22.305886 13689 log.cc:1196] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3e7d7f7ee4384a2ba8bd0dac0ddad757
I20240503 07:59:22.306439 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 3e7d7f7ee4384a2ba8bd0dac0ddad757 (table rebalancer_extra_table_8 [id=914b54d652844204bb800dbd8a5be98a]) successfully deleted
I20240503 07:59:22.386232 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:22.390925 7278 raft_consensus.cc:1261] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:22.390925 9357 raft_consensus.cc:1261] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:22.391430 13405 consensus_queue.cc:1035] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.391657 13406 consensus_queue.cc:1035] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.395922 13729 tablet.cc:1722] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:22.395995 6747 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet 75d347ec0591437bbeb226a6b3096611 (table rebalancer_dynamic_table [id=75456d8f3f454482a616b92801cc178d]). Expected version 1 got 0
I20240503 07:59:22.396433 13729 tablet.cc:1620] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:22.396862 13730 tablet.cc:1722] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:22.397172 13730 tablet.cc:1620] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:22.398155 6740 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet 75d347ec0591437bbeb226a6b3096611 (table rebalancer_dynamic_table [id=75456d8f3f454482a616b92801cc178d]). Expected version 1 got 0
I20240503 07:59:22.399839 13731 tablet.cc:1722] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:22.400185 13731 tablet.cc:1620] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:22.401499 6747 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=75456d8f3f454482a616b92801cc178d] alter complete (version 1)
I20240503 07:59:22.453289 13405 raft_consensus.cc:482] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.453428 13405 raft_consensus.cc:504] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.453820 13405 leader_election.cc:288] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:22.454161 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730eb09f729844ce9f5952c46031c0d8" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:22.454268 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730eb09f729844ce9f5952c46031c0d8" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:22.454438 9767 raft_consensus.cc:2454] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:22.454540 9505 raft_consensus.cc:2454] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:22.454857 9572 leader_election.cc:302] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:22.455150 13405 raft_consensus.cc:2781] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.455219 13405 raft_consensus.cc:482] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.455268 13405 raft_consensus.cc:3037] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.456327 13405 raft_consensus.cc:504] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.456671 13405 leader_election.cc:288] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:22.457019 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730eb09f729844ce9f5952c46031c0d8" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:22.457152 9767 raft_consensus.cc:3037] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.457125 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730eb09f729844ce9f5952c46031c0d8" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:22.457242 9505 raft_consensus.cc:3037] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.458189 9505 raft_consensus.cc:2454] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:22.458189 9767 raft_consensus.cc:2454] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:22.458528 9572 leader_election.cc:302] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:22.458714 13405 raft_consensus.cc:2781] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.458815 13405 raft_consensus.cc:686] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:22.458993 13405 consensus_queue.cc:237] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:22.461179 6738 catalog_manager.cc:5484] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:22.482944 9350 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:22.487071 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:22.494544 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:22.496131 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:22.497655 9746 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:22.498329 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:22.498631 9485 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:22.501502 9615 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:22.505334 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:22.516585 13730 tablet.cc:1722] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:22.516957 13730 tablet.cc:1620] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:22.518445 13729 tablet.cc:1722] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:22.518779 13729 tablet.cc:1620] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:22.525095 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=75456d8f3f454482a616b92801cc178d] alter complete (version 2)
I20240503 07:59:22.527410 13731 tablet.cc:1722] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:22.527801 13731 tablet.cc:1620] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:22.569355 7150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "795f69249afe444ba0828875e1839b8c"
dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34674
I20240503 07:59:22.569558 7150 raft_consensus.cc:595] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:22.614213 13418 consensus_queue.cc:1035] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.619658 13141 consensus_queue.cc:1035] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.636477 6742 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:22.643858 13729 tablet.cc:1722] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:22.645025 13731 tablet.cc:1722] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:22.646112 13730 tablet.cc:1722] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:22.652758 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=75456d8f3f454482a616b92801cc178d] alter complete (version 3)
I20240503 07:59:22.677947 9350 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:22.682088 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:22.687403 12644 raft_consensus.cc:979] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8: : Instructing follower 3befbb2d206f4be89d3e07f38837374b to start an election
I20240503 07:59:22.687525 12644 raft_consensus.cc:1067] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Signalling peer 3befbb2d206f4be89d3e07f38837374b to start an election
I20240503 07:59:22.687680 13591 consensus_queue.cc:1035] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.689844 9374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "795f69249afe444ba0828875e1839b8c"
dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
from {username='slave'} at 127.0.99.131:44395
I20240503 07:59:22.690023 9374 raft_consensus.cc:482] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:22.690094 9374 raft_consensus.cc:3037] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:22.691887 9374 raft_consensus.cc:504] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:22.692252 11859 consensus_queue.cc:1035] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.692332 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:22.692711 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795f69249afe444ba0828875e1839b8c" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:22.692848 7282 raft_consensus.cc:3037] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:22.692966 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795f69249afe444ba0828875e1839b8c" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:22.693102 6886 raft_consensus.cc:3037] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:22.693989 7282 raft_consensus.cc:2454] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 2.
I20240503 07:59:22.694125 6886 raft_consensus.cc:2454] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 2.
I20240503 07:59:22.696506 9307 leader_election.cc:302] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:22.696559 9374 leader_election.cc:288] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 2 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:22.697046 13216 raft_consensus.cc:2781] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:22.697214 13216 raft_consensus.cc:686] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [term 2 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:22.697361 13216 consensus_queue.cc:237] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:22.700923 9485 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:22.700886 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795f69249afe444ba0828875e1839b8c" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:22.701032 7150 raft_consensus.cc:3032] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:22.701068 7150 raft_consensus.cc:729] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:22.701236 7150 consensus_queue.cc:260] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:22.701431 7150 raft_consensus.cc:3037] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:22.702634 7150 raft_consensus.cc:2454] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 2.
I20240503 07:59:22.703363 6745 catalog_manager.cc:5484] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 1 to 2, leader changed from 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 2 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:22.705457 9615 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:22.735319 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:22.740725 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:22.741461 9746 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:22.758518 6747 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:22.758699 6747 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:22.762008 6747 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 75d347ec0591437bbeb226a6b3096611
I20240503 07:59:22.762753 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d347ec0591437bbeb226a6b3096611 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:22.762797 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d347ec0591437bbeb226a6b3096611 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:22.763324 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d347ec0591437bbeb226a6b3096611 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:22.766557 13808 tablet_replica.cc:330] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:22.766780 13808 raft_consensus.cc:2229] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:22.766999 13808 raft_consensus.cc:2258] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.767155 13807 tablet_replica.cc:330] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:22.767366 13809 tablet_replica.cc:330] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:22.767519 13809 raft_consensus.cc:2229] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:22.767691 13807 raft_consensus.cc:2229] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:22.767849 13809 raft_consensus.cc:2258] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.768050 13808 ts_tablet_manager.cc:1875] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.768085 13807 raft_consensus.cc:2258] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.768761 13809 ts_tablet_manager.cc:1875] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.769199 13807 ts_tablet_manager.cc:1875] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.771607 13809 ts_tablet_manager.cc:1888] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.771792 13808 ts_tablet_manager.cc:1888] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.771908 13808 log.cc:1196] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/75d347ec0591437bbeb226a6b3096611
I20240503 07:59:22.772291 13808 ts_tablet_manager.cc:1909] T 75d347ec0591437bbeb226a6b3096611 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:22.773095 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 75d347ec0591437bbeb226a6b3096611 (table rebalancer_dynamic_table_new_name [id=75456d8f3f454482a616b92801cc178d]) successfully deleted
I20240503 07:59:22.774231 13807 ts_tablet_manager.cc:1888] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.774384 13809 log.cc:1196] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/75d347ec0591437bbeb226a6b3096611
I20240503 07:59:22.774668 13809 ts_tablet_manager.cc:1909] T 75d347ec0591437bbeb226a6b3096611 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:22.775319 13807 log.cc:1196] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/75d347ec0591437bbeb226a6b3096611
I20240503 07:59:22.775555 13807 ts_tablet_manager.cc:1909] T 75d347ec0591437bbeb226a6b3096611 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:22.776139 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 75d347ec0591437bbeb226a6b3096611 (table rebalancer_dynamic_table_new_name [id=75456d8f3f454482a616b92801cc178d]) successfully deleted
I20240503 07:59:22.776175 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 75d347ec0591437bbeb226a6b3096611 (table rebalancer_dynamic_table_new_name [id=75456d8f3f454482a616b92801cc178d]) successfully deleted
I20240503 07:59:22.864117 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:22.868188 9350 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:22.869360 13406 consensus_queue.cc:1035] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.869858 7131 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:22.873903 6867 tablet_service.cc:1467] Processing CreateTablet for tablet dd9053c7e1f74f158cf788a00d906a57 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=892d2fdf4327447e9a34a4f517a8c0ed]), partition=
I20240503 07:59:22.873981 9614 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:22.874014 9485 tablet_service.cc:1467] Processing CreateTablet for tablet dd9053c7e1f74f158cf788a00d906a57 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=892d2fdf4327447e9a34a4f517a8c0ed]), partition=
I20240503 07:59:22.874226 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd9053c7e1f74f158cf788a00d906a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.874409 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd9053c7e1f74f158cf788a00d906a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.874598 9615 tablet_service.cc:1467] Processing CreateTablet for tablet dd9053c7e1f74f158cf788a00d906a57 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=892d2fdf4327447e9a34a4f517a8c0ed]), partition=
I20240503 07:59:22.874842 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd9053c7e1f74f158cf788a00d906a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.877559 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:22.880013 13863 tablet_bootstrap.cc:492] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:22.881237 13863 tablet_bootstrap.cc:654] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.883389 13864 tablet_bootstrap.cc:492] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:22.883702 13865 tablet_bootstrap.cc:492] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:22.884893 13863 tablet_bootstrap.cc:492] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:22.885025 13863 ts_tablet_manager.cc:1367] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:22.885677 13865 tablet_bootstrap.cc:654] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.885861 13863 raft_consensus.cc:348] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:22.886011 13863 raft_consensus.cc:374] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.886039 13863 raft_consensus.cc:729] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.886183 13863 consensus_queue.cc:260] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:22.886449 13864 tablet_bootstrap.cc:654] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.887490 13863 ts_tablet_manager.cc:1398] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:22.887637 13864 tablet_bootstrap.cc:492] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:22.887739 13864 ts_tablet_manager.cc:1367] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 07:59:22.888250 13864 raft_consensus.cc:348] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:22.888376 13864 raft_consensus.cc:374] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.888409 13864 raft_consensus.cc:729] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.888525 13864 consensus_queue.cc:260] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:22.888806 13864 ts_tablet_manager.cc:1398] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:22.891075 13789 consensus_queue.cc:1035] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.898612 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:22.902379 9485 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:22.902706 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:22.907428 13865 tablet_bootstrap.cc:492] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:22.907699 13865 ts_tablet_manager.cc:1367] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240503 07:59:22.909062 13865 raft_consensus.cc:348] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:22.910466 13865 raft_consensus.cc:374] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.910584 13865 raft_consensus.cc:729] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.910851 13865 consensus_queue.cc:260] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:22.913429 13865 ts_tablet_manager.cc:1398] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:22.914808 9746 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:22.970424 7020 consensus_queue.cc:237] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:22.971699 6886 raft_consensus.cc:1261] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:22.971758 9374 raft_consensus.cc:1261] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:22.972182 11859 consensus_queue.cc:1035] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:22.972386 13886 consensus_queue.cc:1035] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:22.974270 13886 raft_consensus.cc:2932] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:22.974993 9374 raft_consensus.cc:2932] T 7d3eb284c0fc48b4912168c434844284 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:22.976052 6886 raft_consensus.cc:2932] T 7d3eb284c0fc48b4912168c434844284 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:22.978451 6747 catalog_manager.cc:5484] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:22.984079 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3eb284c0fc48b4912168c434844284 with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:22.984339 13808 tablet_replica.cc:330] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:22.984499 13808 raft_consensus.cc:2229] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:22.984664 13808 raft_consensus.cc:2258] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.985252 13808 ts_tablet_manager.cc:1875] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:22.988324 13808 ts_tablet_manager.cc:1888] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:22.988405 13808 log.cc:1196] T 7d3eb284c0fc48b4912168c434844284 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d3eb284c0fc48b4912168c434844284
I20240503 07:59:22.989784 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 7d3eb284c0fc48b4912168c434844284 (table rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]) successfully deleted
I20240503 07:59:22.994856 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:22.998412 13141 raft_consensus.cc:482] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.998533 13141 raft_consensus.cc:504] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:22.998930 13141 leader_election.cc:288] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:22.999459 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9053c7e1f74f158cf788a00d906a57" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:22.999608 9636 raft_consensus.cc:2454] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:23.000309 9438 leader_election.cc:302] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:23.001818 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 61b93171848746be89447d2ef90459a7 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=667865e90f934944b36fd3b1fd5f68dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.002108 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b93171848746be89447d2ef90459a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.002511 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 05ec40f9eb264e579f8aaf64700390c2 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=667865e90f934944b36fd3b1fd5f68dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.002784 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ec40f9eb264e579f8aaf64700390c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.002969 9614 tablet_service.cc:1467] Processing CreateTablet for tablet 61b93171848746be89447d2ef90459a7 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=667865e90f934944b36fd3b1fd5f68dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.003270 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b93171848746be89447d2ef90459a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.002519 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 61b93171848746be89447d2ef90459a7 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=667865e90f934944b36fd3b1fd5f68dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.003527 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b93171848746be89447d2ef90459a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.004881 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 05ec40f9eb264e579f8aaf64700390c2 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=667865e90f934944b36fd3b1fd5f68dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.005151 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ec40f9eb264e579f8aaf64700390c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.005491 13865 tablet_bootstrap.cc:492] T 05ec40f9eb264e579f8aaf64700390c2 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:23.005657 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 05ec40f9eb264e579f8aaf64700390c2 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=667865e90f934944b36fd3b1fd5f68dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.005872 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ec40f9eb264e579f8aaf64700390c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.006084 13900 tablet_bootstrap.cc:492] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:23.006546 13865 tablet_bootstrap.cc:654] T 05ec40f9eb264e579f8aaf64700390c2 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.007205 13864 tablet_bootstrap.cc:492] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:23.007297 13900 tablet_bootstrap.cc:654] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.007517 13863 tablet_bootstrap.cc:492] T 05ec40f9eb264e579f8aaf64700390c2 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:23.008271 13864 tablet_bootstrap.cc:654] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.008198 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6b23d02dc94e9f84f68203baaf17c2 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=667865e90f934944b36fd3b1fd5f68dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.008387 13900 tablet_bootstrap.cc:492] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:23.008472 13900 ts_tablet_manager.cc:1367] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:23.008502 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6b23d02dc94e9f84f68203baaf17c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.008507 13863 tablet_bootstrap.cc:654] T 05ec40f9eb264e579f8aaf64700390c2 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.008977 13900 raft_consensus.cc:348] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.009090 13900 raft_consensus.cc:374] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.009119 13900 raft_consensus.cc:729] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.009232 13900 consensus_queue.cc:260] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.009531 13900 ts_tablet_manager.cc:1398] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.011796 13905 tablet_bootstrap.cc:492] T 3e6b23d02dc94e9f84f68203baaf17c2 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:23.012724 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6b23d02dc94e9f84f68203baaf17c2 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=667865e90f934944b36fd3b1fd5f68dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.012749 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9053c7e1f74f158cf788a00d906a57" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:23.012900 6886 raft_consensus.cc:2454] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:23.013012 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6b23d02dc94e9f84f68203baaf17c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.013448 13905 tablet_bootstrap.cc:654] T 3e6b23d02dc94e9f84f68203baaf17c2 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.013923 13141 raft_consensus.cc:2781] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.013983 13141 raft_consensus.cc:482] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.014015 13141 raft_consensus.cc:3037] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.014750 13905 tablet_bootstrap.cc:492] T 3e6b23d02dc94e9f84f68203baaf17c2 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:23.014838 13905 ts_tablet_manager.cc:1367] T 3e6b23d02dc94e9f84f68203baaf17c2 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:23.002969 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6b23d02dc94e9f84f68203baaf17c2 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=667865e90f934944b36fd3b1fd5f68dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.015664 13141 raft_consensus.cc:504] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.015821 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6b23d02dc94e9f84f68203baaf17c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.015955 13905 raft_consensus.cc:348] T 3e6b23d02dc94e9f84f68203baaf17c2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:23.016050 13905 raft_consensus.cc:374] T 3e6b23d02dc94e9f84f68203baaf17c2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.016078 13905 raft_consensus.cc:729] T 3e6b23d02dc94e9f84f68203baaf17c2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.016068 13141 leader_election.cc:288] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:23.016223 13905 consensus_queue.cc:260] T 3e6b23d02dc94e9f84f68203baaf17c2 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:23.016567 13905 ts_tablet_manager.cc:1398] T 3e6b23d02dc94e9f84f68203baaf17c2 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:23.018952 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9053c7e1f74f158cf788a00d906a57" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:23.019080 9636 raft_consensus.cc:3037] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.019207 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9053c7e1f74f158cf788a00d906a57" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:23.019337 6886 raft_consensus.cc:3037] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.020043 9636 raft_consensus.cc:2454] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:23.020129 6886 raft_consensus.cc:2454] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:23.020354 9438 leader_election.cc:302] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:23.023214 13141 raft_consensus.cc:2781] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.023322 13141 raft_consensus.cc:686] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:23.023466 13141 consensus_queue.cc:237] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.026046 13864 tablet_bootstrap.cc:492] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:23.026167 13864 ts_tablet_manager.cc:1367] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240503 07:59:23.026255 13863 tablet_bootstrap.cc:492] T 05ec40f9eb264e579f8aaf64700390c2 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:23.026376 13863 ts_tablet_manager.cc:1367] T 05ec40f9eb264e579f8aaf64700390c2 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 07:59:23.026235 6747 catalog_manager.cc:5484] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:23.026705 13864 raft_consensus.cc:348] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.026814 13864 raft_consensus.cc:374] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.026844 13864 raft_consensus.cc:729] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.026824 13863 raft_consensus.cc:348] T 05ec40f9eb264e579f8aaf64700390c2 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.026935 13863 raft_consensus.cc:374] T 05ec40f9eb264e579f8aaf64700390c2 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.026961 13863 raft_consensus.cc:729] T 05ec40f9eb264e579f8aaf64700390c2 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.026957 13864 consensus_queue.cc:260] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.027065 13863 consensus_queue.cc:260] T 05ec40f9eb264e579f8aaf64700390c2 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.027230 13864 ts_tablet_manager.cc:1398] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.027328 13863 ts_tablet_manager.cc:1398] T 05ec40f9eb264e579f8aaf64700390c2 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.027410 13864 tablet_bootstrap.cc:492] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:23.028430 13864 tablet_bootstrap.cc:654] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.034631 13907 tablet_bootstrap.cc:492] T 3e6b23d02dc94e9f84f68203baaf17c2 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:23.036072 13907 tablet_bootstrap.cc:654] T 3e6b23d02dc94e9f84f68203baaf17c2 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.038450 13865 tablet_bootstrap.cc:492] T 05ec40f9eb264e579f8aaf64700390c2 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:23.038584 13865 ts_tablet_manager.cc:1367] T 05ec40f9eb264e579f8aaf64700390c2 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20240503 07:59:23.038906 13864 tablet_bootstrap.cc:492] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:23.039007 13864 ts_tablet_manager.cc:1367] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:23.039091 13865 raft_consensus.cc:348] T 05ec40f9eb264e579f8aaf64700390c2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.039209 13865 raft_consensus.cc:374] T 05ec40f9eb264e579f8aaf64700390c2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.039235 13865 raft_consensus.cc:729] T 05ec40f9eb264e579f8aaf64700390c2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.039367 13865 consensus_queue.cc:260] T 05ec40f9eb264e579f8aaf64700390c2 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.039584 13864 raft_consensus.cc:348] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.039666 13865 ts_tablet_manager.cc:1398] T 05ec40f9eb264e579f8aaf64700390c2 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.039698 13864 raft_consensus.cc:374] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.039728 13864 raft_consensus.cc:729] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.039876 13865 tablet_bootstrap.cc:492] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:23.039893 13864 consensus_queue.cc:260] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.040256 13864 ts_tablet_manager.cc:1398] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.041015 13865 tablet_bootstrap.cc:654] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.041005 13864 tablet_bootstrap.cc:492] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:23.041981 13864 tablet_bootstrap.cc:654] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.042213 13865 tablet_bootstrap.cc:492] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:23.042294 13865 ts_tablet_manager.cc:1367] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:23.042716 13865 raft_consensus.cc:348] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.042821 13865 raft_consensus.cc:374] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.042852 13865 raft_consensus.cc:729] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.042953 13865 consensus_queue.cc:260] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.043237 13865 ts_tablet_manager.cc:1398] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.043551 13907 tablet_bootstrap.cc:492] T 3e6b23d02dc94e9f84f68203baaf17c2 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:23.043663 13907 ts_tablet_manager.cc:1367] T 3e6b23d02dc94e9f84f68203baaf17c2 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:23.044127 13907 raft_consensus.cc:348] T 3e6b23d02dc94e9f84f68203baaf17c2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:23.044327 13907 raft_consensus.cc:374] T 3e6b23d02dc94e9f84f68203baaf17c2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.044407 13907 raft_consensus.cc:729] T 3e6b23d02dc94e9f84f68203baaf17c2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.044617 13907 consensus_queue.cc:260] T 3e6b23d02dc94e9f84f68203baaf17c2 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:23.045091 13907 ts_tablet_manager.cc:1398] T 3e6b23d02dc94e9f84f68203baaf17c2 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:23.051252 13864 tablet_bootstrap.cc:492] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:23.051385 13864 ts_tablet_manager.cc:1367] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:23.051934 13864 raft_consensus.cc:348] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:23.052049 13864 raft_consensus.cc:374] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.052075 13864 raft_consensus.cc:729] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.052189 13864 consensus_queue.cc:260] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:23.052455 13864 ts_tablet_manager.cc:1398] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.078058 6886 raft_consensus.cc:1261] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:23.078959 13216 consensus_queue.cc:1035] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:23.099462 7150 raft_consensus.cc:1261] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:23.100185 13805 consensus_queue.cc:1035] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:23.103682 7282 raft_consensus.cc:1261] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:23.104146 13945 consensus_queue.cc:1035] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:23.124697 13789 raft_consensus.cc:482] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.124861 13789 raft_consensus.cc:504] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:23.125337 13789 leader_election.cc:288] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:23.125649 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6b23d02dc94e9f84f68203baaf17c2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:23.125802 9766 raft_consensus.cc:2454] T 3e6b23d02dc94e9f84f68203baaf17c2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:23.125833 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6b23d02dc94e9f84f68203baaf17c2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:23.125981 7020 raft_consensus.cc:2454] T 3e6b23d02dc94e9f84f68203baaf17c2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:23.126077 9572 leader_election.cc:302] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:23.126252 13789 raft_consensus.cc:2781] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.126312 13789 raft_consensus.cc:482] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.126360 13789 raft_consensus.cc:3037] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.127323 13789 raft_consensus.cc:504] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:23.127663 13789 leader_election.cc:288] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:23.127985 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6b23d02dc94e9f84f68203baaf17c2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:23.128114 9766 raft_consensus.cc:3037] T 3e6b23d02dc94e9f84f68203baaf17c2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.128201 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6b23d02dc94e9f84f68203baaf17c2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:23.128316 7020 raft_consensus.cc:3037] T 3e6b23d02dc94e9f84f68203baaf17c2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.128950 9766 raft_consensus.cc:2454] T 3e6b23d02dc94e9f84f68203baaf17c2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:23.129149 7020 raft_consensus.cc:2454] T 3e6b23d02dc94e9f84f68203baaf17c2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:23.129252 9572 leader_election.cc:302] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:23.129415 13789 raft_consensus.cc:2781] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.129496 13789 raft_consensus.cc:686] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:23.129652 13789 consensus_queue.cc:237] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:23.131626 6746 catalog_manager.cc:5484] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:23.138084 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:23.142522 6886 raft_consensus.cc:1261] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.142585 9636 raft_consensus.cc:1261] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.142947 13141 consensus_queue.cc:1035] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.143115 13418 consensus_queue.cc:1035] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.145840 6746 catalog_manager.cc:5576] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655) does not have the latest schema for tablet dd9053c7e1f74f158cf788a00d906a57 (table rebalancer_dynamic_table [id=892d2fdf4327447e9a34a4f517a8c0ed]). Expected version 1 got 0
I20240503 07:59:23.147933 13955 tablet.cc:1722] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.148126 6743 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet dd9053c7e1f74f158cf788a00d906a57 (table rebalancer_dynamic_table [id=892d2fdf4327447e9a34a4f517a8c0ed]). Expected version 1 got 0
I20240503 07:59:23.148298 13955 tablet.cc:1620] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:23.149459 6746 catalog_manager.cc:5576] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655) does not have the latest schema for tablet dd9053c7e1f74f158cf788a00d906a57 (table rebalancer_dynamic_table [id=892d2fdf4327447e9a34a4f517a8c0ed]). Expected version 1 got 0
I20240503 07:59:23.148823 13954 tablet.cc:1722] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.150844 13954 tablet.cc:1620] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:23.152729 13956 tablet.cc:1722] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.152984 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=892d2fdf4327447e9a34a4f517a8c0ed] alter complete (version 1)
I20240503 07:59:23.153056 13956 tablet.cc:1620] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:23.257177 6743 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:23.261407 13406 raft_consensus.cc:482] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.261540 13406 raft_consensus.cc:504] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.261993 13406 leader_election.cc:288] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:23.262936 13956 tablet.cc:1722] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:23.263144 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ec40f9eb264e579f8aaf64700390c2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:23.263281 13956 tablet.cc:1620] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:23.263293 9505 raft_consensus.cc:2454] T 05ec40f9eb264e579f8aaf64700390c2 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:23.263563 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ec40f9eb264e579f8aaf64700390c2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:23.263693 6888 raft_consensus.cc:2454] T 05ec40f9eb264e579f8aaf64700390c2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:23.263978 9569 leader_election.cc:302] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:23.264158 13406 raft_consensus.cc:2781] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.264219 13406 raft_consensus.cc:482] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.264256 13406 raft_consensus.cc:3037] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.265161 13406 raft_consensus.cc:504] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.265519 13406 leader_election.cc:288] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:23.265975 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ec40f9eb264e579f8aaf64700390c2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:23.266086 6888 raft_consensus.cc:3037] T 05ec40f9eb264e579f8aaf64700390c2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.266366 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ec40f9eb264e579f8aaf64700390c2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:23.266482 9505 raft_consensus.cc:3037] T 05ec40f9eb264e579f8aaf64700390c2 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.267177 6888 raft_consensus.cc:2454] T 05ec40f9eb264e579f8aaf64700390c2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:23.267376 13954 tablet.cc:1722] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:23.267444 9505 raft_consensus.cc:2454] T 05ec40f9eb264e579f8aaf64700390c2 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:23.267692 13954 tablet.cc:1620] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:23.267755 9572 leader_election.cc:302] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:23.268548 13955 tablet.cc:1722] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:23.268837 13955 tablet.cc:1620] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:23.270195 13406 raft_consensus.cc:2781] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.270673 6743 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=892d2fdf4327447e9a34a4f517a8c0ed] alter complete (version 2)
I20240503 07:59:23.271061 13406 raft_consensus.cc:686] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:23.271217 13406 consensus_queue.cc:237] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.273042 6743 catalog_manager.cc:5484] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:23.302819 13945 raft_consensus.cc:482] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.302954 13945 raft_consensus.cc:504] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.303335 13945 leader_election.cc:288] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:23.303709 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b93171848746be89447d2ef90459a7" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:23.303754 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b93171848746be89447d2ef90459a7" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:23.303849 9636 raft_consensus.cc:2454] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:23.303905 6886 raft_consensus.cc:2454] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:23.304140 9307 leader_election.cc:302] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:23.304317 13945 raft_consensus.cc:2781] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.304392 13945 raft_consensus.cc:482] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.304438 13945 raft_consensus.cc:3037] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.305579 13945 raft_consensus.cc:504] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.305917 13945 leader_election.cc:288] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:23.306227 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b93171848746be89447d2ef90459a7" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:23.306380 9636 raft_consensus.cc:3037] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.306376 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b93171848746be89447d2ef90459a7" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:23.306500 6886 raft_consensus.cc:3037] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.307469 6886 raft_consensus.cc:2454] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:23.307489 9636 raft_consensus.cc:2454] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:23.307777 9307 leader_election.cc:302] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:23.307955 13945 raft_consensus.cc:2781] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.308080 13945 raft_consensus.cc:686] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:23.308249 13945 consensus_queue.cc:237] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:23.310261 6747 catalog_manager.cc:5484] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:23.343791 9350 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:23.348517 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:23.348733 7131 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:23.366503 9615 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:23.370487 9485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:23.376111 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:23.376797 6747 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:23.379202 9746 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:23.381027 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:23.387445 13956 tablet.cc:1722] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:23.387624 13954 tablet.cc:1722] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:23.395390 13955 tablet.cc:1722] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:23.401293 6747 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=892d2fdf4327447e9a34a4f517a8c0ed] alter complete (version 3)
I20240503 07:59:23.462039 7282 consensus_queue.cc:237] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:23.463562 9504 raft_consensus.cc:1261] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:23.463600 6888 raft_consensus.cc:1261] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:23.464030 13949 consensus_queue.cc:1035] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:23.464396 12749 consensus_queue.cc:1035] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:23.466249 13564 raft_consensus.cc:2932] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:23.466676 9504 raft_consensus.cc:2932] T aa45918bb38c4583ad59524bda2d48ee P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:23.468425 6888 raft_consensus.cc:2932] T aa45918bb38c4583ad59524bda2d48ee P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:23.470057 6747 catalog_manager.cc:5484] T aa45918bb38c4583ad59524bda2d48ee P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:23.473868 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet aa45918bb38c4583ad59524bda2d48ee with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:23.474263 13992 tablet_replica.cc:330] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:23.474438 13992 raft_consensus.cc:2229] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:23.474624 13992 raft_consensus.cc:2258] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.475209 13992 ts_tablet_manager.cc:1875] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:23.477628 13992 ts_tablet_manager.cc:1888] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:23.477713 13992 log.cc:1196] T aa45918bb38c4583ad59524bda2d48ee P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa45918bb38c4583ad59524bda2d48ee
I20240503 07:59:23.478253 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet aa45918bb38c4583ad59524bda2d48ee (table rebalance_test_table_9 [id=3e242002464141aaa2f77f2115a129ff]) successfully deleted
I20240503 07:59:23.502617 6743 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:23.502791 6743 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:23.505266 6743 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet dd9053c7e1f74f158cf788a00d906a57
I20240503 07:59:23.505928 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9053c7e1f74f158cf788a00d906a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:23 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:23.506103 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9053c7e1f74f158cf788a00d906a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:23 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:23.506106 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9053c7e1f74f158cf788a00d906a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:23 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:23.506531 14005 tablet_replica.cc:330] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:23.506582 14006 tablet_replica.cc:330] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:23.506692 14005 raft_consensus.cc:2229] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:23.506783 14007 tablet_replica.cc:330] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:23.506840 14005 raft_consensus.cc:2258] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.506919 14007 raft_consensus.cc:2229] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:23.507159 14007 raft_consensus.cc:2258] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.507401 14006 raft_consensus.cc:2229] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:23.507539 14006 raft_consensus.cc:2258] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.507577 14005 ts_tablet_manager.cc:1875] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:23.508117 14006 ts_tablet_manager.cc:1875] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:23.509209 14007 ts_tablet_manager.cc:1875] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:23.509974 14005 ts_tablet_manager.cc:1888] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:23.510107 14005 log.cc:1196] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/dd9053c7e1f74f158cf788a00d906a57
I20240503 07:59:23.510476 14005 ts_tablet_manager.cc:1909] T dd9053c7e1f74f158cf788a00d906a57 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:23.511348 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet dd9053c7e1f74f158cf788a00d906a57 (table rebalancer_dynamic_table_new_name [id=892d2fdf4327447e9a34a4f517a8c0ed]) successfully deleted
I20240503 07:59:23.513809 14007 ts_tablet_manager.cc:1888] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:23.513936 14006 ts_tablet_manager.cc:1888] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:23.514024 14006 log.cc:1196] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/dd9053c7e1f74f158cf788a00d906a57
I20240503 07:59:23.514423 14007 log.cc:1196] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/dd9053c7e1f74f158cf788a00d906a57
I20240503 07:59:23.514688 14007 ts_tablet_manager.cc:1909] T dd9053c7e1f74f158cf788a00d906a57 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:23.514973 14006 ts_tablet_manager.cc:1909] T dd9053c7e1f74f158cf788a00d906a57 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:23.515740 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet dd9053c7e1f74f158cf788a00d906a57 (table rebalancer_dynamic_table_new_name [id=892d2fdf4327447e9a34a4f517a8c0ed]) successfully deleted
I20240503 07:59:23.516213 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet dd9053c7e1f74f158cf788a00d906a57 (table rebalancer_dynamic_table_new_name [id=892d2fdf4327447e9a34a4f517a8c0ed]) successfully deleted
I20240503 07:59:23.531371 13406 consensus_queue.cc:1035] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.536363 13789 consensus_queue.cc:1035] T 3e6b23d02dc94e9f84f68203baaf17c2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.577625 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:23.578476 7131 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:23.587805 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:23.593266 9746 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:23.605058 9615 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:23.607398 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:23.609234 9350 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:23.609513 9485 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:23.612962 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:23.617172 9615 tablet_service.cc:1467] Processing CreateTablet for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7dff679e5507454eb75dfdc343993c5a]), partition=
I20240503 07:59:23.617172 9746 tablet_service.cc:1467] Processing CreateTablet for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7dff679e5507454eb75dfdc343993c5a]), partition=
I20240503 07:59:23.617471 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.617472 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.620374 9485 tablet_service.cc:1467] Processing CreateTablet for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7dff679e5507454eb75dfdc343993c5a]), partition=
I20240503 07:59:23.620620 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.645952 14050 tablet_bootstrap.cc:492] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:23.646278 14052 tablet_bootstrap.cc:492] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:23.627511 14051 tablet_bootstrap.cc:492] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:23.648810 14052 tablet_bootstrap.cc:654] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.649842 14050 tablet_bootstrap.cc:654] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.650103 14051 tablet_bootstrap.cc:654] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.659897 14050 tablet_bootstrap.cc:492] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:23.660109 14050 ts_tablet_manager.cc:1367] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:23.661186 14050 raft_consensus.cc:348] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.661346 14050 raft_consensus.cc:374] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.661376 14050 raft_consensus.cc:729] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.661502 14050 consensus_queue.cc:260] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.661769 14050 ts_tablet_manager.cc:1398] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:23.662309 14052 tablet_bootstrap.cc:492] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:23.662442 14052 ts_tablet_manager.cc:1367] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 07:59:23.663978 14052 raft_consensus.cc:348] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.664108 14052 raft_consensus.cc:374] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.664134 14052 raft_consensus.cc:729] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.664237 14052 consensus_queue.cc:260] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.664525 14052 ts_tablet_manager.cc:1398] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:23.668259 14051 tablet_bootstrap.cc:492] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:23.668365 14051 ts_tablet_manager.cc:1367] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.000s
I20240503 07:59:23.669376 14051 raft_consensus.cc:348] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.669585 14051 raft_consensus.cc:374] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.669654 14051 raft_consensus.cc:729] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.669839 14051 consensus_queue.cc:260] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.670936 14051 ts_tablet_manager.cc:1398] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:23.700053 7019 consensus_queue.cc:237] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:23.701437 7282 raft_consensus.cc:1261] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:23.701488 9357 raft_consensus.cc:1261] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:23.701828 14034 consensus_queue.cc:1035] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:23.702018 14038 consensus_queue.cc:1035] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:23.703787 11859 raft_consensus.cc:2932] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:23.704234 7282 raft_consensus.cc:2932] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:23.704241 9357 raft_consensus.cc:2932] T abb55c09be324f37a1a546e73008a80d P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:23.706950 6741 catalog_manager.cc:5484] T abb55c09be324f37a1a546e73008a80d P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:23.709843 7131 tablet_service.cc:1514] Processing DeleteTablet for tablet abb55c09be324f37a1a546e73008a80d with delete_type TABLET_DATA_TOMBSTONED (TS 00a3f8e159c64327832d79fb32cdcca8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:23.710206 14074 tablet_replica.cc:330] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:23.710392 14074 raft_consensus.cc:2229] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:23.710613 14074 raft_consensus.cc:2258] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.711450 14074 ts_tablet_manager.cc:1875] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:23.713968 14074 ts_tablet_manager.cc:1888] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:23.714043 14074 log.cc:1196] T abb55c09be324f37a1a546e73008a80d P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/abb55c09be324f37a1a546e73008a80d
I20240503 07:59:23.714668 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet abb55c09be324f37a1a546e73008a80d (table rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]) successfully deleted
I20240503 07:59:23.758955 6888 consensus_queue.cc:237] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:23.760592 7282 raft_consensus.cc:1261] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.760636 9357 raft_consensus.cc:1261] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:23.760859 6822 consensus_peers.cc:485] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 517c0e477a8c40fbb84c4cd3bb6ecabd. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:23.761034 13087 consensus_queue.cc:1035] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:23.761222 14111 consensus_queue.cc:1035] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:23.763459 13087 raft_consensus.cc:2932] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:23.763895 9357 raft_consensus.cc:2932] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:23.765431 7282 raft_consensus.cc:2932] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:23.767818 6738 catalog_manager.cc:5484] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index -1 to 2, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:23.787444 9504 consensus_queue.cc:237] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:23.788904 9766 raft_consensus.cc:1261] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:23.789058 9441 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): Couldn't send request to peer 5227b26243484bbfb177a995ceb50afc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e051a416bb446df91e90eba13206341. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:23.789252 13141 raft_consensus.cc:482] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.789276 13953 consensus_queue.cc:1035] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:23.789383 13141 raft_consensus.cc:504] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.789531 9357 raft_consensus.cc:1261] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.789688 13141 leader_election.cc:288] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:23.790004 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e22b2a0d964c5cbf1aa00ba9ca8426" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:23.790102 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e22b2a0d964c5cbf1aa00ba9ca8426" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:23.790148 9636 raft_consensus.cc:2454] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:23.790230 9767 raft_consensus.cc:2454] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:23.790535 9441 leader_election.cc:302] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:23.790747 13953 raft_consensus.cc:2781] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.790796 13953 raft_consensus.cc:482] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.790828 13953 raft_consensus.cc:3037] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.789898 13141 consensus_queue.cc:1035] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:23.791954 13953 raft_consensus.cc:504] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.792299 13953 leader_election.cc:288] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:23.792608 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e22b2a0d964c5cbf1aa00ba9ca8426" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:23.792693 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e22b2a0d964c5cbf1aa00ba9ca8426" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:23.792764 9636 raft_consensus.cc:3037] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.792815 9766 raft_consensus.cc:3037] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.792920 13141 raft_consensus.cc:2932] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:23.793388 9357 raft_consensus.cc:2932] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:23.793826 9636 raft_consensus.cc:2454] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:23.793830 9766 raft_consensus.cc:2454] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:23.795440 9438 leader_election.cc:302] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:23.795922 13141 raft_consensus.cc:2781] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.796010 13141 raft_consensus.cc:686] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:23.796142 13141 consensus_queue.cc:237] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.796415 6746 catalog_manager.cc:5484] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:23.796949 9766 raft_consensus.cc:2932] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:23.802953 6746 catalog_manager.cc:5484] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:23.818079 7282 consensus_queue.cc:237] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:23.819882 7150 raft_consensus.cc:1261] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:23.819932 6888 raft_consensus.cc:1261] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:23.820307 13564 consensus_queue.cc:1035] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:23.820502 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 721c34a8b6a348e5890d462ba7da938c. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:23.820926 13564 consensus_queue.cc:1035] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:23.822870 13949 raft_consensus.cc:2932] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:23.823302 6886 raft_consensus.cc:2932] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:23.825505 7150 raft_consensus.cc:2932] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:23.825875 6746 catalog_manager.cc:5484] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:23.828821 13216 consensus_queue.cc:1035] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.829303 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:23.831604 13789 consensus_queue.cc:1035] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.835742 13945 consensus_queue.cc:1035] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.835913 13406 consensus_queue.cc:1035] T 05ec40f9eb264e579f8aaf64700390c2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.840996 9746 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef6defd02954e009565bae98f5e8418 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=ae9c8bb841ff4e139b8fc59887fda0e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.841357 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef6defd02954e009565bae98f5e8418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.841728 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef6defd02954e009565bae98f5e8418 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=ae9c8bb841ff4e139b8fc59887fda0e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.842068 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef6defd02954e009565bae98f5e8418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.842553 9615 tablet_service.cc:1467] Processing CreateTablet for tablet d54f576fbe3d44ff8311505aef9855e2 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=ae9c8bb841ff4e139b8fc59887fda0e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.842821 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d54f576fbe3d44ff8311505aef9855e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.843741 14051 tablet_bootstrap.cc:492] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:23.844794 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 659657d85679433abdcbe711e4741b67 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=ae9c8bb841ff4e139b8fc59887fda0e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.845037 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659657d85679433abdcbe711e4741b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.845064 14051 tablet_bootstrap.cc:654] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.845242 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 659657d85679433abdcbe711e4741b67 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=ae9c8bb841ff4e139b8fc59887fda0e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.845463 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659657d85679433abdcbe711e4741b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.845736 14050 tablet_bootstrap.cc:492] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:23.846228 9353 tablet_service.cc:1467] Processing CreateTablet for tablet d54f576fbe3d44ff8311505aef9855e2 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=ae9c8bb841ff4e139b8fc59887fda0e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.846526 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d54f576fbe3d44ff8311505aef9855e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.846716 14050 tablet_bootstrap.cc:654] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.847286 9745 tablet_service.cc:1467] Processing CreateTablet for tablet d54f576fbe3d44ff8311505aef9855e2 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=ae9c8bb841ff4e139b8fc59887fda0e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.847515 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d54f576fbe3d44ff8311505aef9855e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.847541 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef6defd02954e009565bae98f5e8418 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=ae9c8bb841ff4e139b8fc59887fda0e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.847775 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef6defd02954e009565bae98f5e8418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.848482 14167 tablet_bootstrap.cc:492] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:23.848999 9351 tablet_service.cc:1467] Processing CreateTablet for tablet 659657d85679433abdcbe711e4741b67 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=ae9c8bb841ff4e139b8fc59887fda0e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.849308 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659657d85679433abdcbe711e4741b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.849647 14167 tablet_bootstrap.cc:654] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.850363 14170 tablet_bootstrap.cc:492] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:23.851145 14050 tablet_bootstrap.cc:492] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:23.851238 14050 ts_tablet_manager.cc:1367] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:23.851399 14170 tablet_bootstrap.cc:654] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.851734 14050 raft_consensus.cc:348] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.851840 14050 raft_consensus.cc:374] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.851871 14050 raft_consensus.cc:729] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.851992 14050 consensus_queue.cc:260] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.852283 14050 ts_tablet_manager.cc:1398] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.852461 14050 tablet_bootstrap.cc:492] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:23.852461 14170 tablet_bootstrap.cc:492] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:23.852532 14170 ts_tablet_manager.cc:1367] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:23.852967 14170 raft_consensus.cc:348] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.853085 14170 raft_consensus.cc:374] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.853114 14170 raft_consensus.cc:729] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.853232 14170 consensus_queue.cc:260] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.853348 14050 tablet_bootstrap.cc:654] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.853506 14170 ts_tablet_manager.cc:1398] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.853680 14170 tablet_bootstrap.cc:492] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:23.854609 14170 tablet_bootstrap.cc:654] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.855844 14051 tablet_bootstrap.cc:492] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:23.855939 14051 ts_tablet_manager.cc:1367] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:23.855973 14167 tablet_bootstrap.cc:492] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:23.856060 14167 ts_tablet_manager.cc:1367] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:23.856402 14051 raft_consensus.cc:348] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:23.856511 14051 raft_consensus.cc:374] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.856541 14051 raft_consensus.cc:729] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.856493 14167 raft_consensus.cc:348] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:23.856590 14167 raft_consensus.cc:374] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.856616 14167 raft_consensus.cc:729] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.856660 14051 consensus_queue.cc:260] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:23.856735 14167 consensus_queue.cc:260] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:23.857000 14170 tablet_bootstrap.cc:492] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:23.857030 14167 ts_tablet_manager.cc:1398] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.857060 14170 ts_tablet_manager.cc:1367] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:23.857398 14170 raft_consensus.cc:348] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:23.857512 14170 raft_consensus.cc:374] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.857535 14170 raft_consensus.cc:729] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.857542 14051 ts_tablet_manager.cc:1398] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:23.857656 14170 consensus_queue.cc:260] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:23.857713 14051 tablet_bootstrap.cc:492] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:23.857913 14170 ts_tablet_manager.cc:1398] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.858047 14170 tablet_bootstrap.cc:492] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:23.859220 14170 tablet_bootstrap.cc:654] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.861821 14170 tablet_bootstrap.cc:492] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:23.861910 14170 ts_tablet_manager.cc:1367] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:23.862138 14051 tablet_bootstrap.cc:654] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.862425 14170 raft_consensus.cc:348] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:23.862532 14170 raft_consensus.cc:374] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.862572 14170 raft_consensus.cc:729] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.862684 14170 consensus_queue.cc:260] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:23.862951 14170 ts_tablet_manager.cc:1398] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.863538 14050 tablet_bootstrap.cc:492] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:23.863631 14050 ts_tablet_manager.cc:1367] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:23.864082 14050 raft_consensus.cc:348] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:23.864189 14050 raft_consensus.cc:374] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.864219 14050 raft_consensus.cc:729] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.864353 14050 consensus_queue.cc:260] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:23.864650 14050 ts_tablet_manager.cc:1398] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.867041 14051 tablet_bootstrap.cc:492] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:23.867151 14051 ts_tablet_manager.cc:1367] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 07:59:23.867601 14051 raft_consensus.cc:348] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:23.867725 14051 raft_consensus.cc:374] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.867769 14051 raft_consensus.cc:729] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.867914 14051 consensus_queue.cc:260] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:23.868209 14051 ts_tablet_manager.cc:1398] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.868413 14051 tablet_bootstrap.cc:492] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:23.869449 14051 tablet_bootstrap.cc:654] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.870741 14051 tablet_bootstrap.cc:492] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:23.870883 14051 ts_tablet_manager.cc:1367] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240503 07:59:23.871412 14051 raft_consensus.cc:348] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.871527 14051 raft_consensus.cc:374] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.871578 14051 raft_consensus.cc:729] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.871726 14051 consensus_queue.cc:260] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:23.872157 14051 ts_tablet_manager.cc:1398] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.950716 6743 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:23.955067 9766 raft_consensus.cc:1261] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.955121 9636 raft_consensus.cc:1261] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.955600 14135 consensus_queue.cc:1035] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.955798 14136 consensus_queue.cc:1035] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.958506 6738 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 (table rebalancer_dynamic_table [id=7dff679e5507454eb75dfdc343993c5a]). Expected version 1 got 0
I20240503 07:59:23.958940 6743 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 (table rebalancer_dynamic_table [id=7dff679e5507454eb75dfdc343993c5a]). Expected version 1 got 0
I20240503 07:59:23.959652 13786 mvcc.cc:205] Tried to move back new op lower bound from 7023506079556206592 to 7023506078910279680. Current Snapshot: MvccSnapshot[applied={T|T < 7023506079556206592}]
I20240503 07:59:23.959960 14216 tablet.cc:1722] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.960320 14216 tablet.cc:1620] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:23.960588 14217 tablet.cc:1722] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.960881 14217 tablet.cc:1620] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:23.960875 14218 tablet.cc:1722] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.961918 6743 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 (table rebalancer_dynamic_table [id=7dff679e5507454eb75dfdc343993c5a]). Expected version 1 got 0
I20240503 07:59:23.962123 14218 tablet.cc:1620] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:23.962201 6738 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 (table rebalancer_dynamic_table [id=7dff679e5507454eb75dfdc343993c5a]). Expected version 1 got 0
I20240503 07:59:23.967228 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=7dff679e5507454eb75dfdc343993c5a] alter complete (version 1)
I20240503 07:59:24.029093 13406 raft_consensus.cc:482] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.029219 13406 raft_consensus.cc:504] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:24.029608 13406 leader_election.cc:288] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:24.030011 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d54f576fbe3d44ff8311505aef9855e2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:24.030182 9357 raft_consensus.cc:2454] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:24.030153 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d54f576fbe3d44ff8311505aef9855e2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:24.030292 9766 raft_consensus.cc:2454] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:24.030510 9570 leader_election.cc:302] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:24.030673 13406 raft_consensus.cc:2781] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.030728 13406 raft_consensus.cc:482] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.030768 13406 raft_consensus.cc:3037] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.031821 13406 raft_consensus.cc:504] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:24.032171 13406 leader_election.cc:288] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:24.032510 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d54f576fbe3d44ff8311505aef9855e2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:24.032510 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d54f576fbe3d44ff8311505aef9855e2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:24.032635 9357 raft_consensus.cc:3037] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.032649 9766 raft_consensus.cc:3037] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.033636 9766 raft_consensus.cc:2454] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:24.033636 9357 raft_consensus.cc:2454] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:24.033908 9570 leader_election.cc:302] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:24.034123 13406 raft_consensus.cc:2781] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.034211 13406 raft_consensus.cc:686] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:24.034395 13406 consensus_queue.cc:237] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:24.036621 6738 catalog_manager.cc:5484] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:24.069202 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:24.074766 14216 tablet.cc:1722] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.075151 14216 tablet.cc:1620] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:24.075501 14217 tablet.cc:1722] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.075671 14218 tablet.cc:1722] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.075814 14217 tablet.cc:1620] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:24.076016 14218 tablet.cc:1620] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:24.081401 6744 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=7dff679e5507454eb75dfdc343993c5a] alter complete (version 2)
I20240503 07:59:24.119197 13216 raft_consensus.cc:482] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.119338 13216 raft_consensus.cc:504] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.119732 13216 leader_election.cc:288] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:24.120121 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659657d85679433abdcbe711e4741b67" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:24.120129 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659657d85679433abdcbe711e4741b67" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:24.120268 9767 raft_consensus.cc:2454] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:24.120338 9636 raft_consensus.cc:2454] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:24.120558 9310 leader_election.cc:302] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:24.120738 13216 raft_consensus.cc:2781] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.120795 13216 raft_consensus.cc:482] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.120841 13216 raft_consensus.cc:3037] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.121696 13216 raft_consensus.cc:504] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.122022 13216 leader_election.cc:288] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:24.122326 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659657d85679433abdcbe711e4741b67" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:24.122306 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659657d85679433abdcbe711e4741b67" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:24.122464 9636 raft_consensus.cc:3037] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.122473 9767 raft_consensus.cc:3037] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.123225 9636 raft_consensus.cc:2454] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:24.123286 9767 raft_consensus.cc:2454] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:24.123490 9307 leader_election.cc:302] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:24.123708 13216 raft_consensus.cc:2781] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.123793 13216 raft_consensus.cc:686] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:24.123939 13216 consensus_queue.cc:237] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.125926 6744 catalog_manager.cc:5484] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:24.171753 13216 raft_consensus.cc:482] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.171921 13216 raft_consensus.cc:504] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.172334 13216 leader_election.cc:288] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:24.172690 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6defd02954e009565bae98f5e8418" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:24.172745 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6defd02954e009565bae98f5e8418" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:24.172830 9766 raft_consensus.cc:2454] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:24.172904 7150 raft_consensus.cc:2454] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:24.173219 9310 leader_election.cc:302] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:24.173408 13216 raft_consensus.cc:2781] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.173470 13216 raft_consensus.cc:482] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.173517 13216 raft_consensus.cc:3037] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.174613 13216 raft_consensus.cc:504] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.174958 13216 leader_election.cc:288] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:24.175312 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6defd02954e009565bae98f5e8418" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:24.175346 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6defd02954e009565bae98f5e8418" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:24.175429 9766 raft_consensus.cc:3037] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.175482 7150 raft_consensus.cc:3037] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.176474 9766 raft_consensus.cc:2454] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:24.176564 7150 raft_consensus.cc:2454] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:24.176782 9310 leader_election.cc:302] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:24.176939 13216 raft_consensus.cc:2781] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.177022 13216 raft_consensus.cc:686] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:24.177165 13216 consensus_queue.cc:237] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.179318 6744 catalog_manager.cc:5484] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:24.184315 6744 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:24.190064 14216 tablet.cc:1722] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.190737 14218 tablet.cc:1722] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.191453 14217 tablet.cc:1722] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.197432 6744 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=7dff679e5507454eb75dfdc343993c5a] alter complete (version 3)
I20240503 07:59:24.212507 9351 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:24.219017 9616 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:24.225479 7131 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:24.234513 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:24.243036 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:24.244648 9745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:24.247849 9485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:24.249114 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:24.286335 14238 ts_tablet_manager.cc:902] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc: Initiating tablet copy from peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:24.287335 14238 tablet_copy_client.cc:308] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:45435
I20240503 07:59:24.287905 9525 tablet_copy_service.cc:140] P cb73994515af4880b05383e78368bb4a: Received BeginTabletCopySession request for tablet 6e051a416bb446df91e90eba13206341 from peer 5227b26243484bbfb177a995ceb50afc ({username='slave'} at 127.0.99.132:35461)
I20240503 07:59:24.288053 9525 tablet_copy_service.cc:161] P cb73994515af4880b05383e78368bb4a: Beginning new tablet copy session on tablet 6e051a416bb446df91e90eba13206341 from peer 5227b26243484bbfb177a995ceb50afc at {username='slave'} at 127.0.99.132:35461: session id = 5227b26243484bbfb177a995ceb50afc-6e051a416bb446df91e90eba13206341
I20240503 07:59:24.289263 9525 tablet_copy_source_session.cc:205] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:24.290462 14237 ts_tablet_manager.cc:902] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:24.290992 14238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e051a416bb446df91e90eba13206341. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.292023 14237 tablet_copy_client.cc:308] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:24.292912 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet 721c34a8b6a348e5890d462ba7da938c from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:56405)
I20240503 07:59:24.293015 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet 721c34a8b6a348e5890d462ba7da938c from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:56405: session id = 3befbb2d206f4be89d3e07f38837374b-721c34a8b6a348e5890d462ba7da938c
I20240503 07:59:24.293901 7302 tablet_copy_source_session.cc:205] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:24.294651 14238 tablet_copy_client.cc:783] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:24.294817 14238 tablet_copy_client.cc:647] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:24.294948 14237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721c34a8b6a348e5890d462ba7da938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.297412 14237 tablet_copy_client.cc:783] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:24.297552 14237 tablet_copy_client.cc:647] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:24.298224 14238 tablet_copy_client.cc:523] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:24.299242 14237 tablet_copy_client.cc:523] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:24.299559 6738 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:24.299722 6738 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:24.300058 14238 tablet_bootstrap.cc:492] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:24.300401 14237 tablet_bootstrap.cc:492] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:24.302212 6738 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet d8e22b2a0d964c5cbf1aa00ba9ca8426
I20240503 07:59:24.303329 9745 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:24.303400 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:24.303704 14249 tablet_replica.cc:330] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:24.303895 14249 raft_consensus.cc:2229] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:24.304154 14249 raft_consensus.cc:2258] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.304776 14249 ts_tablet_manager.cc:1875] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.306270 9616 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:24.307233 14249 ts_tablet_manager.cc:1888] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.307343 14249 log.cc:1196] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/d8e22b2a0d964c5cbf1aa00ba9ca8426
I20240503 07:59:24.307616 14249 ts_tablet_manager.cc:1909] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:24.307682 14251 tablet_replica.cc:330] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:24.307847 14251 raft_consensus.cc:2229] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.307983 14251 raft_consensus.cc:2258] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.308491 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 (table rebalancer_dynamic_table_new_name [id=7dff679e5507454eb75dfdc343993c5a]) successfully deleted
I20240503 07:59:24.310302 14251 ts_tablet_manager.cc:1875] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.312994 14251 ts_tablet_manager.cc:1888] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.313162 14251 log.cc:1196] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/d8e22b2a0d964c5cbf1aa00ba9ca8426
I20240503 07:59:24.313534 14251 ts_tablet_manager.cc:1909] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:24.314281 14248 tablet_replica.cc:330] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:24.314286 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 (table rebalancer_dynamic_table_new_name [id=7dff679e5507454eb75dfdc343993c5a]) successfully deleted
I20240503 07:59:24.316097 14238 tablet_bootstrap.cc:492] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:24.316368 14238 tablet_bootstrap.cc:492] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc: Bootstrap complete.
I20240503 07:59:24.316555 14238 ts_tablet_manager.cc:1367] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
I20240503 07:59:24.317071 14238 raft_consensus.cc:348] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:24.317198 14238 raft_consensus.cc:729] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.317332 14238 consensus_queue.cc:260] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:24.317646 14238 ts_tablet_manager.cc:1398] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240503 07:59:24.318081 14248 raft_consensus.cc:2229] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.318226 14248 raft_consensus.cc:2258] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.318928 14248 ts_tablet_manager.cc:1875] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.319983 9525 tablet_copy_service.cc:342] P cb73994515af4880b05383e78368bb4a: Request end of tablet copy session 5227b26243484bbfb177a995ceb50afc-6e051a416bb446df91e90eba13206341 received from {username='slave'} at 127.0.99.132:35461
I20240503 07:59:24.320067 9525 tablet_copy_service.cc:433] P cb73994515af4880b05383e78368bb4a: ending tablet copy session 5227b26243484bbfb177a995ceb50afc-6e051a416bb446df91e90eba13206341 on tablet 6e051a416bb446df91e90eba13206341 with peer 5227b26243484bbfb177a995ceb50afc
I20240503 07:59:24.321416 14248 ts_tablet_manager.cc:1888] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.321560 14248 log.cc:1196] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/d8e22b2a0d964c5cbf1aa00ba9ca8426
I20240503 07:59:24.321899 14248 ts_tablet_manager.cc:1909] T d8e22b2a0d964c5cbf1aa00ba9ca8426 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:24.322666 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet d8e22b2a0d964c5cbf1aa00ba9ca8426 (table rebalancer_dynamic_table_new_name [id=7dff679e5507454eb75dfdc343993c5a]) successfully deleted
I20240503 07:59:24.325346 14237 tablet_bootstrap.cc:492] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:24.325625 14237 tablet_bootstrap.cc:492] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:24.325812 14237 ts_tablet_manager.cc:1367] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.000s
I20240503 07:59:24.326383 14237 raft_consensus.cc:348] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.326525 14237 raft_consensus.cc:729] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.326689 14237 consensus_queue.cc:260] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.326992 14237 ts_tablet_manager.cc:1398] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.327478 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-721c34a8b6a348e5890d462ba7da938c received from {username='slave'} at 127.0.99.133:56405
I20240503 07:59:24.327558 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-721c34a8b6a348e5890d462ba7da938c on tablet 721c34a8b6a348e5890d462ba7da938c with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:24.383548 14293 ts_tablet_manager.cc:902] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:24.386647 14293 tablet_copy_client.cc:308] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:24.387223 6908 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet 517c0e477a8c40fbb84c4cd3bb6ecabd from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:59233)
I20240503 07:59:24.387378 6908 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet 517c0e477a8c40fbb84c4cd3bb6ecabd from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:59233: session id = 159177b3caeb434baa98e04136d050b2-517c0e477a8c40fbb84c4cd3bb6ecabd
I20240503 07:59:24.388471 6908 tablet_copy_source_session.cc:205] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:24.391830 14293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517c0e477a8c40fbb84c4cd3bb6ecabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.396906 14293 tablet_copy_client.cc:783] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:24.397115 14293 tablet_copy_client.cc:647] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:24.400386 14293 tablet_copy_client.cc:523] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:24.402014 14293 tablet_bootstrap.cc:492] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:24.406700 6747 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:24.407243 14293 tablet_bootstrap.cc:492] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:24.407493 14293 tablet_bootstrap.cc:492] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:24.407665 14293 ts_tablet_manager.cc:1367] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 07:59:24.408190 14293 raft_consensus.cc:348] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.408336 14293 raft_consensus.cc:729] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.408476 14293 consensus_queue.cc:260] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.408788 14293 ts_tablet_manager.cc:1398] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:24.410923 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-517c0e477a8c40fbb84c4cd3bb6ecabd received from {username='slave'} at 127.0.99.135:59233
I20240503 07:59:24.411005 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session 159177b3caeb434baa98e04136d050b2-517c0e477a8c40fbb84c4cd3bb6ecabd on tablet 517c0e477a8c40fbb84c4cd3bb6ecabd with peer 159177b3caeb434baa98e04136d050b2
I20240503 07:59:24.416222 9616 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:24.421706 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 345b78f431104d648bb9942fbda97117 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d62d25439dbb4f1c932caa0c16c2813e]), partition=
I20240503 07:59:24.422019 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 345b78f431104d648bb9942fbda97117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.422812 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 345b78f431104d648bb9942fbda97117 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d62d25439dbb4f1c932caa0c16c2813e]), partition=
I20240503 07:59:24.422902 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 345b78f431104d648bb9942fbda97117 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d62d25439dbb4f1c932caa0c16c2813e]), partition=
I20240503 07:59:24.423060 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 345b78f431104d648bb9942fbda97117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.423091 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 345b78f431104d648bb9942fbda97117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.424850 14310 tablet_bootstrap.cc:492] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:24.426899 9351 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:24.427489 14310 tablet_bootstrap.cc:654] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.433389 14312 tablet_bootstrap.cc:492] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:24.436357 14311 tablet_bootstrap.cc:492] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:24.436957 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:24.437064 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:24.437577 14311 tablet_bootstrap.cc:654] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.440292 9745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:24.441080 14312 tablet_bootstrap.cc:654] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.443672 14312 tablet_bootstrap.cc:492] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:24.443794 14312 ts_tablet_manager.cc:1367] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240503 07:59:24.444370 14312 raft_consensus.cc:348] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:24.444510 14312 raft_consensus.cc:374] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.444545 14312 raft_consensus.cc:729] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.444700 14312 consensus_queue.cc:260] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:24.445016 14312 ts_tablet_manager.cc:1398] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:24.445314 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:24.460129 9485 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:24.462020 13789 consensus_queue.cc:1035] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.471223 13789 raft_consensus.cc:482] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.471365 13789 raft_consensus.cc:504] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:24.471750 13789 leader_election.cc:288] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:24.472978 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "345b78f431104d648bb9942fbda97117" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:24.473176 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "345b78f431104d648bb9942fbda97117" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
W20240503 07:59:24.473379 9572 leader_election.cc:341] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Illegal state: must be running to vote when last-logged opid is not known
W20240503 07:59:24.473502 9572 leader_election.cc:341] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:24.473542 9572 leader_election.cc:302] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 159177b3caeb434baa98e04136d050b2; no voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a
I20240503 07:59:24.473771 13789 raft_consensus.cc:2726] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:24.477176 7131 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:24.477191 14311 tablet_bootstrap.cc:492] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:24.477300 14311 ts_tablet_manager.cc:1367] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.000s
I20240503 07:59:24.477859 14311 raft_consensus.cc:348] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:24.477989 14311 raft_consensus.cc:374] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.478020 14311 raft_consensus.cc:729] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.478137 14311 consensus_queue.cc:260] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:24.478860 14311 ts_tablet_manager.cc:1398] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:24.502926 14310 tablet_bootstrap.cc:492] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:24.503057 14310 ts_tablet_manager.cc:1367] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.000s
I20240503 07:59:24.503533 14310 raft_consensus.cc:348] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:24.503659 14310 raft_consensus.cc:374] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.503684 14310 raft_consensus.cc:729] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.505288 14310 consensus_queue.cc:260] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:24.507428 14310 ts_tablet_manager.cc:1398] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:24.516829 13406 consensus_queue.cc:1035] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.547539 13216 consensus_queue.cc:1035] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.552006 13216 consensus_queue.cc:1035] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.553965 14319 raft_consensus.cc:482] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.554100 14319 raft_consensus.cc:504] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:24.554564 14319 leader_election.cc:288] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:24.554951 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "345b78f431104d648bb9942fbda97117" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:24.555083 9636 raft_consensus.cc:2454] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:24.555105 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "345b78f431104d648bb9942fbda97117" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:24.555250 9504 raft_consensus.cc:2454] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:24.555414 9700 leader_election.cc:302] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:24.555779 14319 raft_consensus.cc:2781] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.555841 14319 raft_consensus.cc:482] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.555876 14319 raft_consensus.cc:3037] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.557008 14319 raft_consensus.cc:504] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:24.557365 14319 leader_election.cc:288] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:24.557701 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "345b78f431104d648bb9942fbda97117" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:24.557824 9504 raft_consensus.cc:3037] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.557787 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "345b78f431104d648bb9942fbda97117" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:24.557900 9636 raft_consensus.cc:3037] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.558893 9504 raft_consensus.cc:2454] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:24.558921 9636 raft_consensus.cc:2454] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:24.559218 9703 leader_election.cc:302] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:24.559381 14319 raft_consensus.cc:2781] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.559469 14319 raft_consensus.cc:686] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:24.559638 14319 consensus_queue.cc:237] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:24.561822 6747 catalog_manager.cc:5484] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:24.635152 9745 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:24.635993 9374 raft_consensus.cc:1203] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:24.638816 7131 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:24.639959 9351 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:24.640015 9616 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:24.656215 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:24.665163 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:24.681859 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:24.687099 9485 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:24.689913 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:24.690572 9634 raft_consensus.cc:1261] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:24.691046 14319 consensus_queue.cc:1035] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.693420 9504 raft_consensus.cc:1261] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:24.694787 6741 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 345b78f431104d648bb9942fbda97117 (table rebalancer_dynamic_table [id=d62d25439dbb4f1c932caa0c16c2813e]). Expected version 1 got 0
I20240503 07:59:24.695305 14319 consensus_queue.cc:1035] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.697891 6741 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 345b78f431104d648bb9942fbda97117 (table rebalancer_dynamic_table [id=d62d25439dbb4f1c932caa0c16c2813e]). Expected version 1 got 0
I20240503 07:59:24.712214 6741 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 345b78f431104d648bb9942fbda97117 (table rebalancer_dynamic_table [id=d62d25439dbb4f1c932caa0c16c2813e]). Expected version 1 got 0
I20240503 07:59:24.713946 13216 consensus_queue.cc:1035] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.717414 6738 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 345b78f431104d648bb9942fbda97117 (table rebalancer_dynamic_table [id=d62d25439dbb4f1c932caa0c16c2813e]). Expected version 1 got 0
I20240503 07:59:24.717849 14379 tablet.cc:1722] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:24.718243 14380 tablet.cc:1722] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:24.718730 14379 tablet.cc:1620] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:24.719158 14380 tablet.cc:1620] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:24.722975 14382 tablet.cc:1722] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:24.723291 14382 tablet.cc:1620] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:24.724153 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:24.726028 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=d62d25439dbb4f1c932caa0c16c2813e] alter complete (version 1)
I20240503 07:59:24.730930 9745 tablet_service.cc:1467] Processing CreateTablet for tablet d05ebb7c3aa24804a7aa9a7e1a8e699c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62360cc567504bf19c32fc20e93a496f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.731266 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05ebb7c3aa24804a7aa9a7e1a8e699c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.731518 9351 tablet_service.cc:1467] Processing CreateTablet for tablet ad4d427d99214106ba9ead0ec30b4687 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62360cc567504bf19c32fc20e93a496f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.731787 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4d427d99214106ba9ead0ec30b4687. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.733255 14311 tablet_bootstrap.cc:492] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:24.734365 14311 tablet_bootstrap.cc:654] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.734381 14383 tablet_bootstrap.cc:492] T ad4d427d99214106ba9ead0ec30b4687 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:24.734686 9351 tablet_service.cc:1467] Processing CreateTablet for tablet 96edc0386e0e45a4b9df0f47820c723c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62360cc567504bf19c32fc20e93a496f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.734911 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96edc0386e0e45a4b9df0f47820c723c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.734949 9616 tablet_service.cc:1467] Processing CreateTablet for tablet d05ebb7c3aa24804a7aa9a7e1a8e699c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62360cc567504bf19c32fc20e93a496f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.735231 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05ebb7c3aa24804a7aa9a7e1a8e699c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.735284 7131 tablet_service.cc:1467] Processing CreateTablet for tablet d05ebb7c3aa24804a7aa9a7e1a8e699c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62360cc567504bf19c32fc20e93a496f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.735383 14383 tablet_bootstrap.cc:654] T ad4d427d99214106ba9ead0ec30b4687 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.735566 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05ebb7c3aa24804a7aa9a7e1a8e699c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.736436 14383 tablet_bootstrap.cc:492] T ad4d427d99214106ba9ead0ec30b4687 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:24.736522 14383 ts_tablet_manager.cc:1367] T ad4d427d99214106ba9ead0ec30b4687 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:24.736981 14383 raft_consensus.cc:348] T ad4d427d99214106ba9ead0ec30b4687 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.737092 14383 raft_consensus.cc:374] T ad4d427d99214106ba9ead0ec30b4687 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.737147 14383 raft_consensus.cc:729] T ad4d427d99214106ba9ead0ec30b4687 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.737870 14383 consensus_queue.cc:260] T ad4d427d99214106ba9ead0ec30b4687 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.737991 7129 tablet_service.cc:1467] Processing CreateTablet for tablet ad4d427d99214106ba9ead0ec30b4687 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62360cc567504bf19c32fc20e93a496f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.738232 14383 ts_tablet_manager.cc:1398] T ad4d427d99214106ba9ead0ec30b4687 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:24.738278 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4d427d99214106ba9ead0ec30b4687. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.738464 14383 tablet_bootstrap.cc:492] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:24.739202 14312 tablet_bootstrap.cc:492] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:24.739400 14383 tablet_bootstrap.cc:654] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.740553 14312 tablet_bootstrap.cc:654] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.741294 14383 tablet_bootstrap.cc:492] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:24.741340 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 96edc0386e0e45a4b9df0f47820c723c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62360cc567504bf19c32fc20e93a496f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.741410 14383 ts_tablet_manager.cc:1367] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:24.741580 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96edc0386e0e45a4b9df0f47820c723c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.741641 14312 tablet_bootstrap.cc:492] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:24.741748 14312 ts_tablet_manager.cc:1367] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:24.741930 14383 raft_consensus.cc:348] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.742044 14383 raft_consensus.cc:374] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.742077 14383 raft_consensus.cc:729] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.742223 14383 consensus_queue.cc:260] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.742249 14312 raft_consensus.cc:348] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:24.742336 14312 raft_consensus.cc:374] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.742412 14312 raft_consensus.cc:729] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.742533 14312 consensus_queue.cc:260] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:24.742694 14383 ts_tablet_manager.cc:1398] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.743402 14312 ts_tablet_manager.cc:1398] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:24.743563 14312 tablet_bootstrap.cc:492] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:24.743732 7260 tablet_service.cc:1467] Processing CreateTablet for tablet ad4d427d99214106ba9ead0ec30b4687 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62360cc567504bf19c32fc20e93a496f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.743985 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4d427d99214106ba9ead0ec30b4687. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.744380 14312 tablet_bootstrap.cc:654] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.745329 14312 tablet_bootstrap.cc:492] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:24.745414 14312 ts_tablet_manager.cc:1367] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:24.745965 14390 tablet_bootstrap.cc:492] T ad4d427d99214106ba9ead0ec30b4687 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:24.746214 14312 raft_consensus.cc:348] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.746673 14312 raft_consensus.cc:374] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.746731 14312 raft_consensus.cc:729] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.746893 14312 consensus_queue.cc:260] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.747169 14312 ts_tablet_manager.cc:1398] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:24.747175 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 96edc0386e0e45a4b9df0f47820c723c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62360cc567504bf19c32fc20e93a496f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.747457 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96edc0386e0e45a4b9df0f47820c723c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.748111 14390 tablet_bootstrap.cc:654] T ad4d427d99214106ba9ead0ec30b4687 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.749190 14390 tablet_bootstrap.cc:492] T ad4d427d99214106ba9ead0ec30b4687 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:24.749285 14390 ts_tablet_manager.cc:1367] T ad4d427d99214106ba9ead0ec30b4687 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:24.749708 14310 tablet_bootstrap.cc:492] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:24.750087 14390 raft_consensus.cc:348] T ad4d427d99214106ba9ead0ec30b4687 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.750253 14390 raft_consensus.cc:374] T ad4d427d99214106ba9ead0ec30b4687 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.750290 14390 raft_consensus.cc:729] T ad4d427d99214106ba9ead0ec30b4687 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.750449 7282 raft_consensus.cc:1203] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:24.750680 14310 tablet_bootstrap.cc:654] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.751581 14390 consensus_queue.cc:260] T ad4d427d99214106ba9ead0ec30b4687 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.753523 14310 tablet_bootstrap.cc:492] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:24.753646 14310 ts_tablet_manager.cc:1367] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:24.754122 14310 raft_consensus.cc:348] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.754222 14310 raft_consensus.cc:374] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.754253 14310 raft_consensus.cc:729] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.754392 14310 consensus_queue.cc:260] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.754670 14310 ts_tablet_manager.cc:1398] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.758275 14390 ts_tablet_manager.cc:1398] T ad4d427d99214106ba9ead0ec30b4687 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:24.763826 14311 tablet_bootstrap.cc:492] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:24.764133 14311 ts_tablet_manager.cc:1367] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20240503 07:59:24.765219 14386 tablet_bootstrap.cc:492] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:24.766423 14311 raft_consensus.cc:348] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:24.766568 14311 raft_consensus.cc:374] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.766604 14311 raft_consensus.cc:729] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.766639 14386 tablet_bootstrap.cc:654] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.766763 14311 consensus_queue.cc:260] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:24.767091 14311 ts_tablet_manager.cc:1398] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:24.771958 14222 consensus_queue.cc:1035] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.780347 14386 tablet_bootstrap.cc:492] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:24.780553 14386 ts_tablet_manager.cc:1367] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.002s
I20240503 07:59:24.781280 14386 raft_consensus.cc:348] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:24.781500 14386 raft_consensus.cc:374] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.781582 14386 raft_consensus.cc:729] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.781796 14386 consensus_queue.cc:260] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:24.784610 14386 ts_tablet_manager.cc:1398] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:24.787010 14386 tablet_bootstrap.cc:492] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:24.788105 14386 tablet_bootstrap.cc:654] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.789878 14386 tablet_bootstrap.cc:492] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:24.789968 14386 ts_tablet_manager.cc:1367] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:24.790426 14386 raft_consensus.cc:348] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.790551 14386 raft_consensus.cc:374] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.790607 14386 raft_consensus.cc:729] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.790752 14386 consensus_queue.cc:260] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.791137 14386 ts_tablet_manager.cc:1398] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.812853 9634 raft_consensus.cc:1203] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:24.848065 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:24.855698 14379 tablet.cc:1722] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.856077 14379 tablet.cc:1620] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:24.856243 14382 tablet.cc:1722] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.856559 14382 tablet.cc:1620] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:24.859119 14380 tablet.cc:1722] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.859462 14380 tablet.cc:1620] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:24.861994 13406 raft_consensus.cc:482] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.862104 13406 raft_consensus.cc:504] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.862571 13406 leader_election.cc:288] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:24.862965 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96edc0386e0e45a4b9df0f47820c723c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:24.863143 9374 raft_consensus.cc:2454] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:24.863474 9570 leader_election.cc:302] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:24.863654 13406 raft_consensus.cc:2781] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.863710 13406 raft_consensus.cc:482] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.863754 13406 raft_consensus.cc:3037] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.864282 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96edc0386e0e45a4b9df0f47820c723c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:24.864411 9504 raft_consensus.cc:2454] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:24.864822 13406 raft_consensus.cc:504] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.865168 13406 leader_election.cc:288] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:24.866822 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=d62d25439dbb4f1c932caa0c16c2813e] alter complete (version 2)
I20240503 07:59:24.867954 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96edc0386e0e45a4b9df0f47820c723c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:24.868079 9505 raft_consensus.cc:3037] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.868072 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96edc0386e0e45a4b9df0f47820c723c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:24.868204 9374 raft_consensus.cc:3037] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.869037 9505 raft_consensus.cc:2454] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:24.869339 9374 raft_consensus.cc:2454] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:24.869446 9572 leader_election.cc:302] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:24.870069 13406 raft_consensus.cc:2781] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.870162 13406 raft_consensus.cc:686] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:24.870306 13406 consensus_queue.cc:237] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:24.876019 6739 catalog_manager.cc:5484] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:24.905623 13406 raft_consensus.cc:482] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.905754 13406 raft_consensus.cc:504] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:24.906123 13406 leader_election.cc:288] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:24.906842 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05ebb7c3aa24804a7aa9a7e1a8e699c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:24.907241 9766 raft_consensus.cc:2454] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:24.907946 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05ebb7c3aa24804a7aa9a7e1a8e699c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:24.908191 9572 leader_election.cc:302] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:24.908623 7149 raft_consensus.cc:2454] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:24.908751 13406 raft_consensus.cc:2781] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.908916 13406 raft_consensus.cc:482] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.909066 13406 raft_consensus.cc:3037] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.912290 13406 raft_consensus.cc:504] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:24.912664 13406 leader_election.cc:288] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:24.913100 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05ebb7c3aa24804a7aa9a7e1a8e699c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:24.913148 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05ebb7c3aa24804a7aa9a7e1a8e699c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:24.913228 7149 raft_consensus.cc:3037] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.913295 9766 raft_consensus.cc:3037] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.914119 9766 raft_consensus.cc:2454] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:24.914139 7149 raft_consensus.cc:2454] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:24.914441 9572 leader_election.cc:302] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:24.914681 13406 raft_consensus.cc:2781] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.914803 13406 raft_consensus.cc:686] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:24.914945 13406 consensus_queue.cc:237] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:24.916836 6744 catalog_manager.cc:5484] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:24.949882 14443 raft_consensus.cc:482] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.950027 14443 raft_consensus.cc:504] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.950516 14443 leader_election.cc:288] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:24.950855 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4d427d99214106ba9ead0ec30b4687" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:24.950948 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4d427d99214106ba9ead0ec30b4687" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:24.951013 7282 raft_consensus.cc:2454] T ad4d427d99214106ba9ead0ec30b4687 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:24.951081 9374 raft_consensus.cc:2454] T ad4d427d99214106ba9ead0ec30b4687 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:24.951344 7085 leader_election.cc:302] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:24.951526 14443 raft_consensus.cc:2781] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.951592 14443 raft_consensus.cc:482] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.951628 14443 raft_consensus.cc:3037] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.952827 14443 raft_consensus.cc:504] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.953194 14443 leader_election.cc:288] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:24.953491 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4d427d99214106ba9ead0ec30b4687" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:24.953532 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4d427d99214106ba9ead0ec30b4687" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:24.953627 7282 raft_consensus.cc:3037] T ad4d427d99214106ba9ead0ec30b4687 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.953668 9374 raft_consensus.cc:3037] T ad4d427d99214106ba9ead0ec30b4687 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.954490 9374 raft_consensus.cc:2454] T ad4d427d99214106ba9ead0ec30b4687 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:24.954495 7282 raft_consensus.cc:2454] T ad4d427d99214106ba9ead0ec30b4687 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:24.954771 7085 leader_election.cc:302] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:24.954933 14443 raft_consensus.cc:2781] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.955018 14443 raft_consensus.cc:686] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:24.955155 14443 consensus_queue.cc:237] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:24.957273 6746 catalog_manager.cc:5484] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:24.968979 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:24.982944 14379 tablet.cc:1722] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.983422 7129 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:24.987457 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:24.987623 9615 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:24.988152 14380 tablet.cc:1722] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.990842 14382 tablet.cc:1722] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.992627 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:24.992908 9351 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:25.008812 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=d62d25439dbb4f1c932caa0c16c2813e] alter complete (version 3)
I20240503 07:59:25.012617 9745 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:25.025221 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:25.033000 9485 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:25.087822 6886 consensus_queue.cc:237] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:25.089138 9766 raft_consensus.cc:1261] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:25.089201 7149 raft_consensus.cc:1261] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:25.089584 14472 consensus_queue.cc:1035] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:25.089784 14111 consensus_queue.cc:1035] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:25.091758 14475 raft_consensus.cc:2932] T 3658573037664f03b38fefd31d2fe587 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:25.092191 9766 raft_consensus.cc:2932] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:25.092209 7149 raft_consensus.cc:2932] T 3658573037664f03b38fefd31d2fe587 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:25.095548 6744 catalog_manager.cc:5484] T 3658573037664f03b38fefd31d2fe587 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:25.099627 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 3658573037664f03b38fefd31d2fe587 with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:25.099993 14481 tablet_replica.cc:330] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:25.100550 14481 raft_consensus.cc:2229] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:25.100746 14481 raft_consensus.cc:2258] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:25.101300 14481 ts_tablet_manager.cc:1875] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:25.103542 14481 ts_tablet_manager.cc:1888] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:25.103626 14481 log.cc:1196] T 3658573037664f03b38fefd31d2fe587 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3658573037664f03b38fefd31d2fe587
I20240503 07:59:25.104135 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 3658573037664f03b38fefd31d2fe587 (table rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]) successfully deleted
I20240503 07:59:25.113749 6743 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:25.113924 6743 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:25.116353 6743 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 345b78f431104d648bb9942fbda97117
I20240503 07:59:25.117487 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet 345b78f431104d648bb9942fbda97117 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:25 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:25.117750 9745 tablet_service.cc:1514] Processing DeleteTablet for tablet 345b78f431104d648bb9942fbda97117 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:25 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:25.118158 14489 tablet_replica.cc:330] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:25.118394 14489 raft_consensus.cc:2229] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:25.118640 14489 raft_consensus.cc:2258] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:25.119354 14489 ts_tablet_manager.cc:1875] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:25.119683 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet 345b78f431104d648bb9942fbda97117 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:25 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:25.119978 14490 tablet_replica.cc:330] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:25.120174 14490 raft_consensus.cc:2229] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:25.120338 14490 raft_consensus.cc:2258] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:25.120963 14490 ts_tablet_manager.cc:1875] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:25.121850 14489 ts_tablet_manager.cc:1888] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:25.121968 14489 log.cc:1196] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/345b78f431104d648bb9942fbda97117
I20240503 07:59:25.122277 14489 ts_tablet_manager.cc:1909] T 345b78f431104d648bb9942fbda97117 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:25.122331 14488 tablet_replica.cc:330] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:25.122542 14488 raft_consensus.cc:2229] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:25.122674 14488 raft_consensus.cc:2258] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:25.123106 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 345b78f431104d648bb9942fbda97117 (table rebalancer_dynamic_table_new_name [id=d62d25439dbb4f1c932caa0c16c2813e]) successfully deleted
I20240503 07:59:25.123641 14490 ts_tablet_manager.cc:1888] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:25.123705 14490 log.cc:1196] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/345b78f431104d648bb9942fbda97117
I20240503 07:59:25.123927 14490 ts_tablet_manager.cc:1909] T 345b78f431104d648bb9942fbda97117 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:25.124521 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 345b78f431104d648bb9942fbda97117 (table rebalancer_dynamic_table_new_name [id=d62d25439dbb4f1c932caa0c16c2813e]) successfully deleted
I20240503 07:59:25.127061 14488 ts_tablet_manager.cc:1875] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:25.129585 14488 ts_tablet_manager.cc:1888] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:25.129673 14488 log.cc:1196] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/345b78f431104d648bb9942fbda97117
I20240503 07:59:25.129884 14488 ts_tablet_manager.cc:1909] T 345b78f431104d648bb9942fbda97117 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:25.130632 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 345b78f431104d648bb9942fbda97117 (table rebalancer_dynamic_table_new_name [id=d62d25439dbb4f1c932caa0c16c2813e]) successfully deleted
I20240503 07:59:25.200476 9615 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:25.201260 7129 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:25.203827 9351 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:25.207614 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:25.216953 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:25.218427 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:25.223280 6744 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:25.231598 9351 tablet_service.cc:1467] Processing CreateTablet for tablet 4f62a302e1d840b0a9047596ccde6987 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d051f8b9c6144db9b0410c7cbc17855a]), partition=
I20240503 07:59:25.231652 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 4f62a302e1d840b0a9047596ccde6987 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d051f8b9c6144db9b0410c7cbc17855a]), partition=
I20240503 07:59:25.231900 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f62a302e1d840b0a9047596ccde6987. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.231900 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f62a302e1d840b0a9047596ccde6987. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.234305 14310 tablet_bootstrap.cc:492] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:25.234481 14383 tablet_bootstrap.cc:492] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:25.235396 14383 tablet_bootstrap.cc:654] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.235396 14310 tablet_bootstrap.cc:654] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.236238 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 4f62a302e1d840b0a9047596ccde6987 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d051f8b9c6144db9b0410c7cbc17855a]), partition=
I20240503 07:59:25.236498 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f62a302e1d840b0a9047596ccde6987. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.237390 14310 tablet_bootstrap.cc:492] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:25.237500 14310 ts_tablet_manager.cc:1367] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:25.238040 14310 raft_consensus.cc:348] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:25.238173 14310 raft_consensus.cc:374] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.238199 14310 raft_consensus.cc:729] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.238322 14310 consensus_queue.cc:260] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:25.238641 14310 ts_tablet_manager.cc:1398] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.239279 9485 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:25.239953 9747 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:25.241215 14311 tablet_bootstrap.cc:492] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:25.242270 14311 tablet_bootstrap.cc:654] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.249871 14311 tablet_bootstrap.cc:492] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:25.249979 14311 ts_tablet_manager.cc:1367] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:25.250486 14311 raft_consensus.cc:348] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:25.250597 14311 raft_consensus.cc:374] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.250622 14311 raft_consensus.cc:729] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.250747 14311 consensus_queue.cc:260] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:25.251037 14311 ts_tablet_manager.cc:1398] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.287354 14383 tablet_bootstrap.cc:492] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:25.287492 14383 ts_tablet_manager.cc:1367] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.001s
I20240503 07:59:25.288640 14383 raft_consensus.cc:348] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:25.288777 14383 raft_consensus.cc:374] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.288815 14383 raft_consensus.cc:729] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.288936 14383 consensus_queue.cc:260] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:25.289263 14383 ts_tablet_manager.cc:1398] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:25.322906 9505 consensus_queue.cc:237] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:25.324151 7282 raft_consensus.cc:1261] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:25.324254 9374 raft_consensus.cc:1261] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:25.324781 14551 consensus_queue.cc:1035] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:25.324965 14135 consensus_queue.cc:1035] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:25.326766 14551 raft_consensus.cc:2932] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:25.327456 7282 raft_consensus.cc:2932] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:25.330821 6745 catalog_manager.cc:5484] T df569029c1514bc5ae4d77327cd03478 P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:25.330910 9374 raft_consensus.cc:2932] T df569029c1514bc5ae4d77327cd03478 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:25.334583 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet df569029c1514bc5ae4d77327cd03478 with delete_type TABLET_DATA_TOMBSTONED (TS 159177b3caeb434baa98e04136d050b2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:25.335086 14490 tablet_replica.cc:330] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:25.335263 14490 raft_consensus.cc:2229] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:25.335381 14490 raft_consensus.cc:2258] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:25.335758 14490 ts_tablet_manager.cc:1875] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:25.337697 14490 ts_tablet_manager.cc:1888] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:25.337754 14490 log.cc:1196] T df569029c1514bc5ae4d77327cd03478 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/df569029c1514bc5ae4d77327cd03478
I20240503 07:59:25.338203 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet df569029c1514bc5ae4d77327cd03478 (table rebalancer_extra_table_10 [id=fa3c77fefb014f88b3cad9f4ea921534]) successfully deleted
I20240503 07:59:25.360056 12644 consensus_queue.cc:1035] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.364245 14443 consensus_queue.cc:1035] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.365820 13406 consensus_queue.cc:1035] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.371388 13789 consensus_queue.cc:1035] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.379264 13789 consensus_queue.cc:1035] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.384414 13406 consensus_queue.cc:1035] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.498757 14551 raft_consensus.cc:482] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.498915 14551 raft_consensus.cc:504] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:25.499346 14551 leader_election.cc:288] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:25.499786 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f62a302e1d840b0a9047596ccde6987" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:25.499786 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f62a302e1d840b0a9047596ccde6987" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:25.499944 9357 raft_consensus.cc:2454] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:25.499944 9766 raft_consensus.cc:2454] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:25.500270 9439 leader_election.cc:302] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:25.500447 14551 raft_consensus.cc:2781] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.500505 14551 raft_consensus.cc:482] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.500530 14551 raft_consensus.cc:3037] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.501405 14551 raft_consensus.cc:504] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:25.501746 14551 leader_election.cc:288] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:25.502076 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f62a302e1d840b0a9047596ccde6987" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:25.502209 9766 raft_consensus.cc:3037] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.503209 9766 raft_consensus.cc:2454] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:25.503470 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f62a302e1d840b0a9047596ccde6987" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:25.503509 9441 leader_election.cc:302] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:25.503603 9357 raft_consensus.cc:3037] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.503664 14551 raft_consensus.cc:2781] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.503734 14551 raft_consensus.cc:686] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:25.503870 14551 consensus_queue.cc:237] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:25.504673 9357 raft_consensus.cc:2454] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:25.507078 6741 catalog_manager.cc:5484] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:25.511350 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:25.517469 9745 tablet_service.cc:1467] Processing CreateTablet for tablet cf3d2c6d70bb49fcb1dd23fc0d402796 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eba63fcb11c54ebf8beaa0fe0e7a772a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.517812 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3d2c6d70bb49fcb1dd23fc0d402796. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.517948 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2c21fc7842455f8617cb0405becbcf (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eba63fcb11c54ebf8beaa0fe0e7a772a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.518079 9616 tablet_service.cc:1467] Processing CreateTablet for tablet dc3367cc41234d52b74eee94d6e67069 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eba63fcb11c54ebf8beaa0fe0e7a772a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.518232 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2c21fc7842455f8617cb0405becbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.518319 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3367cc41234d52b74eee94d6e67069. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.519865 14311 tablet_bootstrap.cc:492] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:25.520090 9485 tablet_service.cc:1467] Processing CreateTablet for tablet cf3d2c6d70bb49fcb1dd23fc0d402796 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eba63fcb11c54ebf8beaa0fe0e7a772a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.520327 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3d2c6d70bb49fcb1dd23fc0d402796. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.520864 14311 tablet_bootstrap.cc:654] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.520932 14310 tablet_bootstrap.cc:492] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:25.520933 14606 tablet_bootstrap.cc:492] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:25.521821 14310 tablet_bootstrap.cc:654] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.521857 14606 tablet_bootstrap.cc:654] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.522764 9615 tablet_service.cc:1467] Processing CreateTablet for tablet cf3d2c6d70bb49fcb1dd23fc0d402796 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eba63fcb11c54ebf8beaa0fe0e7a772a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.522997 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3d2c6d70bb49fcb1dd23fc0d402796. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.523120 14310 tablet_bootstrap.cc:492] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:25.523205 14310 ts_tablet_manager.cc:1367] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:25.523687 14310 raft_consensus.cc:348] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:25.523793 14310 raft_consensus.cc:374] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.523823 14310 raft_consensus.cc:729] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.523939 14310 consensus_queue.cc:260] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:25.524219 14310 ts_tablet_manager.cc:1398] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.524377 14310 tablet_bootstrap.cc:492] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:25.525239 14310 tablet_bootstrap.cc:654] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.526789 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2c21fc7842455f8617cb0405becbcf (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eba63fcb11c54ebf8beaa0fe0e7a772a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.526788 9745 tablet_service.cc:1467] Processing CreateTablet for tablet dc3367cc41234d52b74eee94d6e67069 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eba63fcb11c54ebf8beaa0fe0e7a772a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.527048 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3367cc41234d52b74eee94d6e67069. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.527060 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2c21fc7842455f8617cb0405becbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.529493 14611 tablet_bootstrap.cc:492] T 4f2c21fc7842455f8617cb0405becbcf P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:25.530053 14310 tablet_bootstrap.cc:492] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:25.530197 14310 ts_tablet_manager.cc:1367] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:25.530261 9351 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2c21fc7842455f8617cb0405becbcf (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eba63fcb11c54ebf8beaa0fe0e7a772a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.530417 9350 tablet_service.cc:1467] Processing CreateTablet for tablet dc3367cc41234d52b74eee94d6e67069 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eba63fcb11c54ebf8beaa0fe0e7a772a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.530576 14611 tablet_bootstrap.cc:654] T 4f2c21fc7842455f8617cb0405becbcf P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.530591 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2c21fc7842455f8617cb0405becbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.530701 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3367cc41234d52b74eee94d6e67069. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.530813 14310 raft_consensus.cc:348] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:25.530917 14310 raft_consensus.cc:374] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.530948 14310 raft_consensus.cc:729] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.531091 14310 consensus_queue.cc:260] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:25.531371 14310 ts_tablet_manager.cc:1398] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:25.532789 14383 tablet_bootstrap.cc:492] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:25.533027 14606 tablet_bootstrap.cc:492] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:25.533120 14606 ts_tablet_manager.cc:1367] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:25.533607 14383 tablet_bootstrap.cc:654] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.533617 14606 raft_consensus.cc:348] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:25.533735 14606 raft_consensus.cc:374] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.533768 14606 raft_consensus.cc:729] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.534101 14606 consensus_queue.cc:260] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:25.535902 14383 tablet_bootstrap.cc:492] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:25.535993 14383 ts_tablet_manager.cc:1367] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:25.536104 14311 tablet_bootstrap.cc:492] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:25.536188 14311 ts_tablet_manager.cc:1367] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 07:59:25.536470 14383 raft_consensus.cc:348] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:25.536597 14383 raft_consensus.cc:374] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.536644 14383 raft_consensus.cc:729] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.536618 14311 raft_consensus.cc:348] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:25.536715 14311 raft_consensus.cc:374] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.536743 14311 raft_consensus.cc:729] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.536757 14383 consensus_queue.cc:260] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:25.536847 14311 consensus_queue.cc:260] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:25.535439 14606 ts_tablet_manager.cc:1398] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:25.537005 14383 ts_tablet_manager.cc:1398] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.537098 14606 tablet_bootstrap.cc:492] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:25.537111 14311 ts_tablet_manager.cc:1398] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.537165 14383 tablet_bootstrap.cc:492] T 4f2c21fc7842455f8617cb0405becbcf P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:25.537259 14311 tablet_bootstrap.cc:492] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:25.537950 14606 tablet_bootstrap.cc:654] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.538058 14383 tablet_bootstrap.cc:654] T 4f2c21fc7842455f8617cb0405becbcf P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.538058 14311 tablet_bootstrap.cc:654] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.539259 14611 tablet_bootstrap.cc:492] T 4f2c21fc7842455f8617cb0405becbcf P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:25.539398 14611 ts_tablet_manager.cc:1367] T 4f2c21fc7842455f8617cb0405becbcf P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:25.540467 14606 tablet_bootstrap.cc:492] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:25.540599 14606 ts_tablet_manager.cc:1367] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:25.541102 14606 raft_consensus.cc:348] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:25.541440 14606 raft_consensus.cc:374] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.541522 14606 raft_consensus.cc:729] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.541703 14606 consensus_queue.cc:260] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:25.542049 14606 ts_tablet_manager.cc:1398] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.542052 14611 raft_consensus.cc:348] T 4f2c21fc7842455f8617cb0405becbcf P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:25.542232 14611 raft_consensus.cc:374] T 4f2c21fc7842455f8617cb0405becbcf P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.542306 14611 raft_consensus.cc:729] T 4f2c21fc7842455f8617cb0405becbcf P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.542534 14611 consensus_queue.cc:260] T 4f2c21fc7842455f8617cb0405becbcf P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:25.542886 14611 ts_tablet_manager.cc:1398] T 4f2c21fc7842455f8617cb0405becbcf P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:25.544374 14311 tablet_bootstrap.cc:492] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:25.544484 14311 ts_tablet_manager.cc:1367] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 07:59:25.544934 14311 raft_consensus.cc:348] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:25.545042 14311 raft_consensus.cc:374] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.545073 14311 raft_consensus.cc:729] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.545197 14311 consensus_queue.cc:260] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:25.545460 14311 ts_tablet_manager.cc:1398] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:25.548703 14383 tablet_bootstrap.cc:492] T 4f2c21fc7842455f8617cb0405becbcf P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:25.548897 14383 ts_tablet_manager.cc:1367] T 4f2c21fc7842455f8617cb0405becbcf P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 07:59:25.549391 14383 raft_consensus.cc:348] T 4f2c21fc7842455f8617cb0405becbcf P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:25.549548 14383 raft_consensus.cc:374] T 4f2c21fc7842455f8617cb0405becbcf P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.549609 14383 raft_consensus.cc:729] T 4f2c21fc7842455f8617cb0405becbcf P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.550092 14383 consensus_queue.cc:260] T 4f2c21fc7842455f8617cb0405becbcf P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:25.550628 14383 ts_tablet_manager.cc:1398] T 4f2c21fc7842455f8617cb0405becbcf P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:25.556062 9615 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:25.563199 7129 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:25.563637 9351 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:25.566758 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:25.578835 14554 raft_consensus.cc:482] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.578964 14554 raft_consensus.cc:504] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:25.579429 14554 leader_election.cc:288] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:25.579703 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2c21fc7842455f8617cb0405becbcf" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:25.579793 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2c21fc7842455f8617cb0405becbcf" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:25.579847 9357 raft_consensus.cc:2454] T 4f2c21fc7842455f8617cb0405becbcf P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:25.579933 7019 raft_consensus.cc:2454] T 4f2c21fc7842455f8617cb0405becbcf P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:25.580245 9440 leader_election.cc:302] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:25.580511 14554 raft_consensus.cc:2781] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.580565 14554 raft_consensus.cc:482] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.580596 14554 raft_consensus.cc:3037] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.581806 14554 raft_consensus.cc:504] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:25.582154 14554 leader_election.cc:288] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:25.586534 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:25.587517 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2c21fc7842455f8617cb0405becbcf" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:25.587664 7019 raft_consensus.cc:3037] T 4f2c21fc7842455f8617cb0405becbcf P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.588759 7019 raft_consensus.cc:2454] T 4f2c21fc7842455f8617cb0405becbcf P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:25.589247 9440 leader_election.cc:302] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:25.589262 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2c21fc7842455f8617cb0405becbcf" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:25.589391 9357 raft_consensus.cc:3037] T 4f2c21fc7842455f8617cb0405becbcf P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.589435 14554 raft_consensus.cc:2781] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.589747 14554 raft_consensus.cc:686] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:25.589907 14554 consensus_queue.cc:237] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:25.590703 9357 raft_consensus.cc:2454] T 4f2c21fc7842455f8617cb0405becbcf P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:25.592538 6739 catalog_manager.cc:5484] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:25.598618 9745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:25.613682 9485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:25.616622 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:25.650905 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:25.657361 9766 raft_consensus.cc:1261] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:25.657980 9374 raft_consensus.cc:1261] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:25.658015 14554 consensus_queue.cc:1035] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.659468 14554 consensus_queue.cc:1035] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.661598 6739 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 4f62a302e1d840b0a9047596ccde6987 (table rebalancer_dynamic_table [id=d051f8b9c6144db9b0410c7cbc17855a]). Expected version 1 got 0
I20240503 07:59:25.667807 14121 raft_consensus.cc:482] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.667933 14121 raft_consensus.cc:504] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:25.667979 6741 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 4f62a302e1d840b0a9047596ccde6987 (table rebalancer_dynamic_table [id=d051f8b9c6144db9b0410c7cbc17855a]). Expected version 1 got 0
I20240503 07:59:25.668308 14121 leader_election.cc:288] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:25.668704 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3d2c6d70bb49fcb1dd23fc0d402796" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:25.668851 9505 raft_consensus.cc:2454] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:25.668853 6739 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet 4f62a302e1d840b0a9047596ccde6987 (table rebalancer_dynamic_table [id=d051f8b9c6144db9b0410c7cbc17855a]). Expected version 1 got 0
I20240503 07:59:25.669337 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3d2c6d70bb49fcb1dd23fc0d402796" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:25.669484 9636 raft_consensus.cc:2454] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:25.670650 9703 leader_election.cc:302] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:25.672744 14121 raft_consensus.cc:2781] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.672818 14121 raft_consensus.cc:482] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.672854 14121 raft_consensus.cc:3037] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.673275 6739 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet 4f62a302e1d840b0a9047596ccde6987 (table rebalancer_dynamic_table [id=d051f8b9c6144db9b0410c7cbc17855a]). Expected version 1 got 0
I20240503 07:59:25.673990 14121 raft_consensus.cc:504] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:25.674423 14121 leader_election.cc:288] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:25.674880 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3d2c6d70bb49fcb1dd23fc0d402796" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:25.675012 9636 raft_consensus.cc:3037] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.675745 14632 tablet.cc:1722] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:25.675736 14634 tablet.cc:1722] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:25.676105 14632 tablet.cc:1620] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:25.676107 9636 raft_consensus.cc:2454] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:25.676157 14634 tablet.cc:1620] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:25.677889 14633 tablet.cc:1722] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:25.679606 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3d2c6d70bb49fcb1dd23fc0d402796" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:25.679739 9505 raft_consensus.cc:3037] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.680748 9505 raft_consensus.cc:2454] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:25.681756 9700 leader_election.cc:302] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:25.684441 14633 tablet.cc:1620] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:25.686581 14121 raft_consensus.cc:2781] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.686800 14121 raft_consensus.cc:686] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:25.686959 14121 consensus_queue.cc:237] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:25.692778 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=d051f8b9c6144db9b0410c7cbc17855a] alter complete (version 1)
I20240503 07:59:25.696748 6741 catalog_manager.cc:5484] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:25.712458 9374 consensus_queue.cc:237] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:25.713970 9505 raft_consensus.cc:1261] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:25.713987 9766 raft_consensus.cc:1261] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:25.714577 14222 consensus_queue.cc:1035] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:25.714962 14222 consensus_queue.cc:1035] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:25.717332 14655 raft_consensus.cc:2932] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:25.717409 9505 raft_consensus.cc:2932] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:25.718026 9766 raft_consensus.cc:2932] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:25.722002 6745 catalog_manager.cc:5484] T 75f59b9fe4a54e7d8beabce7589800b1 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index 2 to 3, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:25.725878 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f59b9fe4a54e7d8beabce7589800b1 with delete_type TABLET_DATA_TOMBSTONED (TS c4218904dd7e42d2be95b901a6e5db92 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:25.726285 14656 tablet_replica.cc:330] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:25.726598 14656 raft_consensus.cc:2229] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:25.726940 14656 raft_consensus.cc:2258] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:25.727541 14656 ts_tablet_manager.cc:1875] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:25.730641 14656 ts_tablet_manager.cc:1888] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:25.730829 14656 log.cc:1196] T 75f59b9fe4a54e7d8beabce7589800b1 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/75f59b9fe4a54e7d8beabce7589800b1
I20240503 07:59:25.731570 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 75f59b9fe4a54e7d8beabce7589800b1 (table rebalancer_extra_table_8 [id=914b54d652844204bb800dbd8a5be98a]) successfully deleted
I20240503 07:59:25.795477 6742 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:25.801604 14633 tablet.cc:1722] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:25.801966 14633 tablet.cc:1620] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:25.802863 14634 tablet.cc:1722] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:25.802909 14632 tablet.cc:1722] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:25.803189 14634 tablet.cc:1620] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:25.803210 14632 tablet.cc:1620] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:25.811043 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=d051f8b9c6144db9b0410c7cbc17855a] alter complete (version 2)
I20240503 07:59:25.833127 14655 raft_consensus.cc:482] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.833274 14655 raft_consensus.cc:504] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:25.833722 14655 leader_election.cc:288] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:25.834206 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3367cc41234d52b74eee94d6e67069" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:25.834267 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3367cc41234d52b74eee94d6e67069" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:25.834403 9766 raft_consensus.cc:2454] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:25.834424 9636 raft_consensus.cc:2454] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:25.834707 9307 leader_election.cc:302] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:25.834897 14655 raft_consensus.cc:2781] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.834954 14655 raft_consensus.cc:482] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.834988 14655 raft_consensus.cc:3037] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.836179 14655 raft_consensus.cc:504] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:25.836542 14655 leader_election.cc:288] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:25.836922 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3367cc41234d52b74eee94d6e67069" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:25.837019 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3367cc41234d52b74eee94d6e67069" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:25.837069 9636 raft_consensus.cc:3037] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.837145 9766 raft_consensus.cc:3037] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.838196 9766 raft_consensus.cc:2454] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:25.838232 9636 raft_consensus.cc:2454] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:25.838596 9310 leader_election.cc:302] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:25.838761 14655 raft_consensus.cc:2781] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.838862 14655 raft_consensus.cc:686] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:25.839010 14655 consensus_queue.cc:237] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:25.843276 6746 catalog_manager.cc:5484] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:25.886394 9351 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:25.888392 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:25.891861 7129 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:25.893555 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:25.900343 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:25.902222 9615 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:25.911661 9745 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:25.913517 6744 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:25.921108 14633 tablet.cc:1722] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:25.922466 14632 tablet.cc:1722] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:25.925240 14634 tablet.cc:1722] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:25.927771 9485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:25.930656 6744 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=d051f8b9c6144db9b0410c7cbc17855a] alter complete (version 3)
I20240503 07:59:26.000511 9374 consensus_queue.cc:237] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:26.002002 7278 raft_consensus.cc:1261] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:26.002434 6886 raft_consensus.cc:1261] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:26.002406 13216 consensus_queue.cc:1035] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:26.004158 14655 raft_consensus.cc:2932] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:26.004647 7278 raft_consensus.cc:2932] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:26.006678 13216 consensus_queue.cc:1035] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:26.007833 6739 catalog_manager.cc:5484] T 795f69249afe444ba0828875e1839b8c P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index 4 to 6, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:26.008206 6886 raft_consensus.cc:2932] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:26.011148 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 795f69249afe444ba0828875e1839b8c with delete_type TABLET_DATA_TOMBSTONED (TS 00a3f8e159c64327832d79fb32cdcca8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:26.013339 14763 tablet_replica.cc:330] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:26.013512 14763 raft_consensus.cc:2229] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:26.013651 14763 raft_consensus.cc:2258] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.014263 14763 ts_tablet_manager.cc:1875] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:26.018438 14763 ts_tablet_manager.cc:1888] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 07:59:26.018535 14763 log.cc:1196] T 795f69249afe444ba0828875e1839b8c P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/795f69249afe444ba0828875e1839b8c
I20240503 07:59:26.019136 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 795f69249afe444ba0828875e1839b8c (table rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]) successfully deleted
I20240503 07:59:26.040763 6746 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:26.040940 6746 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:26.043879 6746 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 4f62a302e1d840b0a9047596ccde6987
I20240503 07:59:26.044524 9351 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f62a302e1d840b0a9047596ccde6987 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:26 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:26.044704 9745 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f62a302e1d840b0a9047596ccde6987 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:26 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:26.044731 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f62a302e1d840b0a9047596ccde6987 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:26 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:26.045408 14784 tablet_replica.cc:330] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:26.045476 14786 tablet_replica.cc:330] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:26.045643 14784 raft_consensus.cc:2229] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:26.045805 14784 raft_consensus.cc:2258] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.045817 14786 raft_consensus.cc:2229] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:26.046031 14785 tablet_replica.cc:330] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:26.046020 14786 raft_consensus.cc:2258] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.046250 14785 raft_consensus.cc:2229] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:26.046449 14785 raft_consensus.cc:2258] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.046548 14784 ts_tablet_manager.cc:1875] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:26.046790 14786 ts_tablet_manager.cc:1875] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:26.048395 14785 ts_tablet_manager.cc:1875] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:26.049094 14784 ts_tablet_manager.cc:1888] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:26.049166 14784 log.cc:1196] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4f62a302e1d840b0a9047596ccde6987
I20240503 07:59:26.049234 14786 ts_tablet_manager.cc:1888] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:26.049316 14786 log.cc:1196] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/4f62a302e1d840b0a9047596ccde6987
I20240503 07:59:26.049485 14784 ts_tablet_manager.cc:1909] T 4f62a302e1d840b0a9047596ccde6987 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:26.049571 14786 ts_tablet_manager.cc:1909] T 4f62a302e1d840b0a9047596ccde6987 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:26.050223 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 4f62a302e1d840b0a9047596ccde6987 (table rebalancer_dynamic_table_new_name [id=d051f8b9c6144db9b0410c7cbc17855a]) successfully deleted
I20240503 07:59:26.050302 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 4f62a302e1d840b0a9047596ccde6987 (table rebalancer_dynamic_table_new_name [id=d051f8b9c6144db9b0410c7cbc17855a]) successfully deleted
I20240503 07:59:26.051345 14785 ts_tablet_manager.cc:1888] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:26.051426 14785 log.cc:1196] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/4f62a302e1d840b0a9047596ccde6987
I20240503 07:59:26.051684 14785 ts_tablet_manager.cc:1909] T 4f62a302e1d840b0a9047596ccde6987 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:26.052323 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 4f62a302e1d840b0a9047596ccde6987 (table rebalancer_dynamic_table_new_name [id=d051f8b9c6144db9b0410c7cbc17855a]) successfully deleted
I20240503 07:59:26.063545 9636 consensus_queue.cc:237] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:26.065043 9766 raft_consensus.cc:1261] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.065047 9374 raft_consensus.cc:1261] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:26.065263 9572 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): Couldn't send request to peer 5227b26243484bbfb177a995ceb50afc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ac0bb7b2c95434aa38f0f0e3718e89c. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.065560 13406 consensus_queue.cc:1035] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.065742 13789 consensus_queue.cc:1035] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.067441 13789 raft_consensus.cc:2932] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:26.067867 9766 raft_consensus.cc:2932] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:26.069733 9374 raft_consensus.cc:2932] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:26.071511 6739 catalog_manager.cc:5484] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:26.077168 14554 consensus_queue.cc:1035] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.081543 14554 consensus_queue.cc:1035] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.094980 9636 consensus_queue.cc:237] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:26.096552 9505 raft_consensus.cc:1261] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.096591 7019 raft_consensus.cc:1261] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:26.097030 9569 consensus_peers.cc:485] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 -> Peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Couldn't send request to peer c4218904dd7e42d2be95b901a6e5db92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23759e1c50b9402980461599687904b6. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.097195 14798 consensus_queue.cc:1035] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.097488 14798 consensus_queue.cc:1035] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.099215 14798 raft_consensus.cc:2932] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.099669 7019 raft_consensus.cc:2932] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.102567 9505 raft_consensus.cc:2932] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.102613 6746 catalog_manager.cc:5484] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.123399 7278 consensus_queue.cc:237] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
W20240503 07:59:26.124872 7218 consensus_peers.cc:485] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc -> Peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): Couldn't send request to peer 00a3f8e159c64327832d79fb32cdcca8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50169f7d514d4fe3a4f846faaed03325. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.124907 7019 raft_consensus.cc:1261] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.125707 9766 raft_consensus.cc:1261] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.125931 14839 consensus_queue.cc:1035] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.126188 14440 consensus_queue.cc:1035] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.128262 14839 raft_consensus.cc:2932] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:26.128670 9767 raft_consensus.cc:2932] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:26.131583 6746 catalog_manager.cc:5484] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:26.132042 7019 raft_consensus.cc:2932] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:26.146209 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:26.151579 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9a601328d7447b9cdb75999ae153c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b3249c1d06c4b73824b65c169bc9c95]), partition=
I20240503 07:59:26.151894 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9a601328d7447b9cdb75999ae153c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.151894 9351 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9a601328d7447b9cdb75999ae153c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b3249c1d06c4b73824b65c169bc9c95]), partition=
I20240503 07:59:26.152155 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9a601328d7447b9cdb75999ae153c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.154150 6886 consensus_queue.cc:237] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.154302 14857 tablet_bootstrap.cc:492] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:26.154773 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9a601328d7447b9cdb75999ae153c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b3249c1d06c4b73824b65c169bc9c95]), partition=
I20240503 07:59:26.155018 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9a601328d7447b9cdb75999ae153c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.156235 7019 raft_consensus.cc:1261] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:26.156350 7278 raft_consensus.cc:1261] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:26.156705 14860 consensus_queue.cc:1035] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:26.156909 14629 consensus_queue.cc:1035] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:26.157639 6822 consensus_peers.cc:485] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb2190946fb04a6392097f9588ffda6b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.157961 14857 tablet_bootstrap.cc:654] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.158694 14861 tablet_bootstrap.cc:492] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:26.159008 14858 tablet_bootstrap.cc:492] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:26.159802 14861 tablet_bootstrap.cc:654] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.160337 14858 tablet_bootstrap.cc:654] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.160842 14861 tablet_bootstrap.cc:492] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:26.160938 14861 ts_tablet_manager.cc:1367] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:26.161360 14858 tablet_bootstrap.cc:492] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:26.161463 14858 ts_tablet_manager.cc:1367] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:26.161420 14861 raft_consensus.cc:348] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:26.161546 14861 raft_consensus.cc:374] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.161589 14861 raft_consensus.cc:729] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.161720 14861 consensus_queue.cc:260] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:26.161926 14858 raft_consensus.cc:348] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:26.162079 14858 raft_consensus.cc:374] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.162298 14858 raft_consensus.cc:729] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.162513 14858 consensus_queue.cc:260] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:26.162827 14861 ts_tablet_manager.cc:1398] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.162968 14858 ts_tablet_manager.cc:1398] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.163646 14857 tablet_bootstrap.cc:492] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:26.163751 14857 ts_tablet_manager.cc:1367] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240503 07:59:26.164227 14857 raft_consensus.cc:348] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:26.164444 14857 raft_consensus.cc:374] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.164531 14857 raft_consensus.cc:729] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.164731 14857 consensus_queue.cc:260] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:26.164665 14629 raft_consensus.cc:2932] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:26.165134 14857 ts_tablet_manager.cc:1398] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.167126 7278 raft_consensus.cc:2932] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:26.169643 6746 catalog_manager.cc:5484] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:26.178635 7019 raft_consensus.cc:2932] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:26.195911 7278 consensus_queue.cc:237] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:26.197433 9374 raft_consensus.cc:1261] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:26.197623 7217 consensus_peers.cc:485] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc -> Peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): Couldn't send request to peer 9c085a3241f14cb7ba1332b508f4b57d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4737c826ebb64177a718d83b939c3e7b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.197875 14839 consensus_queue.cc:1035] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.198422 9767 raft_consensus.cc:1261] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.198895 14839 consensus_queue.cc:1035] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.199795 14121 consensus_queue.cc:1035] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.200923 14839 raft_consensus.cc:2932] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:26.201236 9767 raft_consensus.cc:2932] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:26.204566 6744 catalog_manager.cc:5484] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:26.208801 9374 raft_consensus.cc:2932] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:26.212713 14121 consensus_queue.cc:1035] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.217245 13216 consensus_queue.cc:1035] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.221318 14655 consensus_queue.cc:1035] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.235400 6886 consensus_queue.cc:237] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
W20240503 07:59:26.236960 6825 consensus_peers.cc:485] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 -> Peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Couldn't send request to peer a372e98a670f49e9adca46ab5c836c4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf8e0a0da8b941e28e797de8f8b7a3f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.237216 9636 raft_consensus.cc:1261] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.237385 7149 raft_consensus.cc:1261] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.237628 14868 consensus_queue.cc:1035] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.237844 14860 consensus_queue.cc:1035] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.240017 14868 raft_consensus.cc:2932] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:26.240510 9636 raft_consensus.cc:2932] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:26.243263 6744 catalog_manager.cc:5484] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 2, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:26.245443 7149 raft_consensus.cc:2932] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:26.267433 6886 consensus_queue.cc:237] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:26.269111 7019 raft_consensus.cc:1261] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:26.269192 7278 raft_consensus.cc:1261] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:26.269284 6825 consensus_peers.cc:485] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f365955974634bc58e8a0da0fc2563d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.269490 14860 consensus_queue.cc:1035] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:26.269677 14475 consensus_queue.cc:1035] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:26.271569 14860 raft_consensus.cc:2932] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:26.272029 7278 raft_consensus.cc:2932] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:26.273716 7019 raft_consensus.cc:2932] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:26.275370 6744 catalog_manager.cc:5484] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:26.297900 7278 consensus_queue.cc:237] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.299481 7149 raft_consensus.cc:1261] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:26.299481 7020 raft_consensus.cc:1261] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:26.299572 7215 consensus_peers.cc:485] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fcc4c2eeaec4b359108ec64a6e20da8. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.299890 13949 consensus_queue.cc:1035] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:26.300087 14440 consensus_queue.cc:1035] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:26.301898 14440 raft_consensus.cc:2932] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:26.302388 7020 raft_consensus.cc:2932] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:26.304407 7149 raft_consensus.cc:2932] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:26.305285 6743 catalog_manager.cc:5484] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:26.360018 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:26.367156 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae94f901c2d48968b87e091c2e085e4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=edef03085b2549d8aea4a5d5e4228ee0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.367532 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ae94f901c2d48968b87e091c2e085e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.368297 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae94f901c2d48968b87e091c2e085e4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=edef03085b2549d8aea4a5d5e4228ee0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.368353 9616 tablet_service.cc:1467] Processing CreateTablet for tablet fe38557414dd4f509604884bb23cc8ee (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=edef03085b2549d8aea4a5d5e4228ee0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.368566 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ae94f901c2d48968b87e091c2e085e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.369036 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe38557414dd4f509604884bb23cc8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.370457 9351 tablet_service.cc:1467] Processing CreateTablet for tablet fe38557414dd4f509604884bb23cc8ee (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=edef03085b2549d8aea4a5d5e4228ee0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.370741 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe38557414dd4f509604884bb23cc8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.372248 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae94f901c2d48968b87e091c2e085e4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=edef03085b2549d8aea4a5d5e4228ee0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.372527 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ae94f901c2d48968b87e091c2e085e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.373016 14858 tablet_bootstrap.cc:492] T fe38557414dd4f509604884bb23cc8ee P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:26.374004 14858 tablet_bootstrap.cc:654] T fe38557414dd4f509604884bb23cc8ee P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.375241 14987 tablet_bootstrap.cc:492] T 3ae94f901c2d48968b87e091c2e085e4 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:26.377244 6867 tablet_service.cc:1467] Processing CreateTablet for tablet fe4d550eda29472d8c9ae494d25efc83 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=edef03085b2549d8aea4a5d5e4228ee0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.377902 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4d550eda29472d8c9ae494d25efc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.378321 14987 tablet_bootstrap.cc:654] T 3ae94f901c2d48968b87e091c2e085e4 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.379127 9351 tablet_service.cc:1467] Processing CreateTablet for tablet fe4d550eda29472d8c9ae494d25efc83 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=edef03085b2549d8aea4a5d5e4228ee0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.379482 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4d550eda29472d8c9ae494d25efc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.380100 14858 tablet_bootstrap.cc:492] T fe38557414dd4f509604884bb23cc8ee P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:26.380184 14858 ts_tablet_manager.cc:1367] T fe38557414dd4f509604884bb23cc8ee P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 07:59:26.380657 14858 raft_consensus.cc:348] T fe38557414dd4f509604884bb23cc8ee P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.380764 14858 raft_consensus.cc:374] T fe38557414dd4f509604884bb23cc8ee P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.380801 14858 raft_consensus.cc:729] T fe38557414dd4f509604884bb23cc8ee P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.380911 14858 consensus_queue.cc:260] T fe38557414dd4f509604884bb23cc8ee P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.381194 14858 ts_tablet_manager.cc:1398] T fe38557414dd4f509604884bb23cc8ee P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:26.382123 14987 tablet_bootstrap.cc:492] T 3ae94f901c2d48968b87e091c2e085e4 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:26.382259 14987 ts_tablet_manager.cc:1367] T 3ae94f901c2d48968b87e091c2e085e4 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 07:59:26.382474 14858 tablet_bootstrap.cc:492] T fe4d550eda29472d8c9ae494d25efc83 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:26.383496 14858 tablet_bootstrap.cc:654] T fe4d550eda29472d8c9ae494d25efc83 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.384159 14987 raft_consensus.cc:348] T 3ae94f901c2d48968b87e091c2e085e4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:26.384338 14987 raft_consensus.cc:374] T 3ae94f901c2d48968b87e091c2e085e4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.384373 9745 tablet_service.cc:1467] Processing CreateTablet for tablet fe38557414dd4f509604884bb23cc8ee (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=edef03085b2549d8aea4a5d5e4228ee0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.384459 9747 tablet_service.cc:1467] Processing CreateTablet for tablet fe4d550eda29472d8c9ae494d25efc83 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=edef03085b2549d8aea4a5d5e4228ee0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.384656 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe38557414dd4f509604884bb23cc8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.384858 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4d550eda29472d8c9ae494d25efc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.385033 14987 raft_consensus.cc:729] T 3ae94f901c2d48968b87e091c2e085e4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.385210 14987 consensus_queue.cc:260] T 3ae94f901c2d48968b87e091c2e085e4 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:26.387554 14857 tablet_bootstrap.cc:492] T fe4d550eda29472d8c9ae494d25efc83 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:26.388569 14857 tablet_bootstrap.cc:654] T fe4d550eda29472d8c9ae494d25efc83 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.390578 14987 ts_tablet_manager.cc:1398] T 3ae94f901c2d48968b87e091c2e085e4 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:26.391007 14857 tablet_bootstrap.cc:492] T fe4d550eda29472d8c9ae494d25efc83 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:26.391093 14857 ts_tablet_manager.cc:1367] T fe4d550eda29472d8c9ae494d25efc83 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 07:59:26.391544 14857 raft_consensus.cc:348] T fe4d550eda29472d8c9ae494d25efc83 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:26.391651 14857 raft_consensus.cc:374] T fe4d550eda29472d8c9ae494d25efc83 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.391683 14857 raft_consensus.cc:729] T fe4d550eda29472d8c9ae494d25efc83 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.391803 14857 consensus_queue.cc:260] T fe4d550eda29472d8c9ae494d25efc83 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:26.392586 14857 ts_tablet_manager.cc:1398] T fe4d550eda29472d8c9ae494d25efc83 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.392755 14857 tablet_bootstrap.cc:492] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:26.393652 14857 tablet_bootstrap.cc:654] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.395129 14857 tablet_bootstrap.cc:492] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:26.395211 14857 ts_tablet_manager.cc:1367] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:26.395629 14857 raft_consensus.cc:348] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.395732 14857 raft_consensus.cc:374] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.395764 14857 raft_consensus.cc:729] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.395876 14857 consensus_queue.cc:260] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.396332 14857 ts_tablet_manager.cc:1398] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:26.396859 14858 tablet_bootstrap.cc:492] T fe4d550eda29472d8c9ae494d25efc83 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:26.396950 14858 ts_tablet_manager.cc:1367] T fe4d550eda29472d8c9ae494d25efc83 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20240503 07:59:26.397689 14858 raft_consensus.cc:348] T fe4d550eda29472d8c9ae494d25efc83 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:26.397812 14858 raft_consensus.cc:374] T fe4d550eda29472d8c9ae494d25efc83 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.397845 14858 raft_consensus.cc:729] T fe4d550eda29472d8c9ae494d25efc83 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.397966 14858 consensus_queue.cc:260] T fe4d550eda29472d8c9ae494d25efc83 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:26.398238 14858 ts_tablet_manager.cc:1398] T fe4d550eda29472d8c9ae494d25efc83 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:26.406577 14984 tablet_bootstrap.cc:492] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:26.408418 14984 tablet_bootstrap.cc:654] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.409292 14985 tablet_bootstrap.cc:492] T fe38557414dd4f509604884bb23cc8ee P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:26.410614 14985 tablet_bootstrap.cc:654] T fe38557414dd4f509604884bb23cc8ee P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.411587 14984 tablet_bootstrap.cc:492] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:26.411691 14984 ts_tablet_manager.cc:1367] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:26.412832 14984 raft_consensus.cc:348] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:26.413523 14985 tablet_bootstrap.cc:492] T fe38557414dd4f509604884bb23cc8ee P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:26.413627 14984 raft_consensus.cc:374] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.413718 14984 raft_consensus.cc:729] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.414263 14985 ts_tablet_manager.cc:1367] T fe38557414dd4f509604884bb23cc8ee P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:26.414439 14984 consensus_queue.cc:260] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:26.414817 14121 raft_consensus.cc:482] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.414943 14121 raft_consensus.cc:504] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:26.415520 14121 leader_election.cc:288] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:26.416317 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9a601328d7447b9cdb75999ae153c9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:26.416461 9504 raft_consensus.cc:2454] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:26.416796 9703 leader_election.cc:302] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:26.417268 14121 raft_consensus.cc:2781] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.417330 14121 raft_consensus.cc:482] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.417364 14121 raft_consensus.cc:3037] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.417349 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9a601328d7447b9cdb75999ae153c9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:26.417491 9374 raft_consensus.cc:2454] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:26.418567 14121 raft_consensus.cc:504] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:26.418983 14121 leader_election.cc:288] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:26.419672 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9a601328d7447b9cdb75999ae153c9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:26.419796 9374 raft_consensus.cc:3037] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.419729 14985 raft_consensus.cc:348] T fe38557414dd4f509604884bb23cc8ee P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.419850 14985 raft_consensus.cc:374] T fe38557414dd4f509604884bb23cc8ee P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.419878 14985 raft_consensus.cc:729] T fe38557414dd4f509604884bb23cc8ee P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.420104 14984 ts_tablet_manager.cc:1398] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:26.420902 14984 tablet_bootstrap.cc:492] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:26.420926 9374 raft_consensus.cc:2454] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:26.421669 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9a601328d7447b9cdb75999ae153c9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:26.421806 9504 raft_consensus.cc:3037] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.421944 14985 consensus_queue.cc:260] T fe38557414dd4f509604884bb23cc8ee P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.422324 14985 ts_tablet_manager.cc:1398] T fe38557414dd4f509604884bb23cc8ee P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:26.422623 9701 leader_election.cc:302] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:26.422971 9504 raft_consensus.cc:2454] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:26.423005 14121 raft_consensus.cc:2781] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.423100 14121 raft_consensus.cc:686] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:26.423230 14121 consensus_queue.cc:237] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:26.423425 14985 tablet_bootstrap.cc:492] T 3ae94f901c2d48968b87e091c2e085e4 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:26.424441 14984 tablet_bootstrap.cc:654] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.426700 6742 catalog_manager.cc:5484] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:26.426877 14985 tablet_bootstrap.cc:654] T 3ae94f901c2d48968b87e091c2e085e4 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.428017 14984 tablet_bootstrap.cc:492] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:26.428162 14984 ts_tablet_manager.cc:1367] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:26.428668 14984 raft_consensus.cc:348] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:26.428763 14984 raft_consensus.cc:374] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.428791 14984 raft_consensus.cc:729] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.428927 14984 consensus_queue.cc:260] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:26.429358 14984 ts_tablet_manager.cc:1398] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.432830 14985 tablet_bootstrap.cc:492] T 3ae94f901c2d48968b87e091c2e085e4 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:26.432991 14985 ts_tablet_manager.cc:1367] T 3ae94f901c2d48968b87e091c2e085e4 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 07:59:26.434515 14985 raft_consensus.cc:348] T 3ae94f901c2d48968b87e091c2e085e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:26.434727 14985 raft_consensus.cc:374] T 3ae94f901c2d48968b87e091c2e085e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.434851 14985 raft_consensus.cc:729] T 3ae94f901c2d48968b87e091c2e085e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.435063 14985 consensus_queue.cc:260] T 3ae94f901c2d48968b87e091c2e085e4 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:26.438534 14985 ts_tablet_manager.cc:1398] T 3ae94f901c2d48968b87e091c2e085e4 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:26.479485 9615 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:26.482475 15010 ts_tablet_manager.cc:902] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92: Initiating tablet copy from peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:26.485271 15010 tablet_copy_client.cc:308] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:43369
I20240503 07:59:26.485953 9656 tablet_copy_service.cc:140] P 159177b3caeb434baa98e04136d050b2: Received BeginTabletCopySession request for tablet 23759e1c50b9402980461599687904b6 from peer c4218904dd7e42d2be95b901a6e5db92 ({username='slave'} at 127.0.99.129:40095)
I20240503 07:59:26.486093 9656 tablet_copy_service.cc:161] P 159177b3caeb434baa98e04136d050b2: Beginning new tablet copy session on tablet 23759e1c50b9402980461599687904b6 from peer c4218904dd7e42d2be95b901a6e5db92 at {username='slave'} at 127.0.99.129:40095: session id = c4218904dd7e42d2be95b901a6e5db92-23759e1c50b9402980461599687904b6
I20240503 07:59:26.487572 9656 tablet_copy_source_session.cc:205] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:26.489094 15010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23759e1c50b9402980461599687904b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.490514 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:26.491012 7129 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:26.493196 15010 tablet_copy_client.cc:783] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:26.493383 15010 tablet_copy_client.cc:647] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:26.495229 15010 tablet_copy_client.cc:523] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:26.496809 15010 tablet_bootstrap.cc:492] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:26.498389 9351 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:26.498574 6867 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:26.500276 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:26.502990 15010 tablet_bootstrap.cc:492] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:26.503235 15010 tablet_bootstrap.cc:492] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap complete.
I20240503 07:59:26.503381 15010 ts_tablet_manager.cc:1367] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 07:59:26.503851 15010 raft_consensus.cc:348] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:26.503968 15010 raft_consensus.cc:729] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.504083 15010 consensus_queue.cc:260] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:26.504529 15010 ts_tablet_manager.cc:1398] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:26.512917 9745 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:26.520808 9656 tablet_copy_service.cc:342] P 159177b3caeb434baa98e04136d050b2: Request end of tablet copy session c4218904dd7e42d2be95b901a6e5db92-23759e1c50b9402980461599687904b6 received from {username='slave'} at 127.0.99.129:40095
I20240503 07:59:26.520937 9656 tablet_copy_service.cc:433] P 159177b3caeb434baa98e04136d050b2: ending tablet copy session c4218904dd7e42d2be95b901a6e5db92-23759e1c50b9402980461599687904b6 on tablet 23759e1c50b9402980461599687904b6 with peer c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:26.530953 9485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:26.534111 15017 ts_tablet_manager.cc:902] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc: Initiating tablet copy from peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:26.554968 15017 tablet_copy_client.cc:308] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:43369
I20240503 07:59:26.555557 9656 tablet_copy_service.cc:140] P 159177b3caeb434baa98e04136d050b2: Received BeginTabletCopySession request for tablet 1ac0bb7b2c95434aa38f0f0e3718e89c from peer 5227b26243484bbfb177a995ceb50afc ({username='slave'} at 127.0.99.132:56797)
I20240503 07:59:26.555676 9656 tablet_copy_service.cc:161] P 159177b3caeb434baa98e04136d050b2: Beginning new tablet copy session on tablet 1ac0bb7b2c95434aa38f0f0e3718e89c from peer 5227b26243484bbfb177a995ceb50afc at {username='slave'} at 127.0.99.132:56797: session id = 5227b26243484bbfb177a995ceb50afc-1ac0bb7b2c95434aa38f0f0e3718e89c
I20240503 07:59:26.556638 9656 tablet_copy_source_session.cc:205] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:26.558957 15017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac0bb7b2c95434aa38f0f0e3718e89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.561455 15017 tablet_copy_client.cc:783] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:26.561638 15017 tablet_copy_client.cc:647] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:26.561759 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:26.562772 15017 tablet_copy_client.cc:523] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:26.564149 15017 tablet_bootstrap.cc:492] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:26.566291 9504 raft_consensus.cc:1261] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.566731 14635 consensus_queue.cc:1035] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.567248 9374 raft_consensus.cc:1261] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.567696 14121 consensus_queue.cc:1035] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.568365 15017 tablet_bootstrap.cc:492] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:26.568648 15017 tablet_bootstrap.cc:492] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc: Bootstrap complete.
I20240503 07:59:26.568847 15017 ts_tablet_manager.cc:1367] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:26.569519 15017 raft_consensus.cc:348] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:26.569707 15017 raft_consensus.cc:729] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.569854 6740 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 8b9a601328d7447b9cdb75999ae153c9 (table rebalancer_dynamic_table [id=9b3249c1d06c4b73824b65c169bc9c95]). Expected version 1 got 0
I20240503 07:59:26.569913 15017 consensus_queue.cc:260] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:26.570523 6744 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet 8b9a601328d7447b9cdb75999ae153c9 (table rebalancer_dynamic_table [id=9b3249c1d06c4b73824b65c169bc9c95]). Expected version 1 got 0
I20240503 07:59:26.571323 15017 ts_tablet_manager.cc:1398] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.571842 9656 tablet_copy_service.cc:342] P 159177b3caeb434baa98e04136d050b2: Request end of tablet copy session 5227b26243484bbfb177a995ceb50afc-1ac0bb7b2c95434aa38f0f0e3718e89c received from {username='slave'} at 127.0.99.132:56797
I20240503 07:59:26.571923 9656 tablet_copy_service.cc:433] P 159177b3caeb434baa98e04136d050b2: ending tablet copy session 5227b26243484bbfb177a995ceb50afc-1ac0bb7b2c95434aa38f0f0e3718e89c on tablet 1ac0bb7b2c95434aa38f0f0e3718e89c with peer 5227b26243484bbfb177a995ceb50afc
I20240503 07:59:26.576803 14475 raft_consensus.cc:482] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.576916 14475 raft_consensus.cc:504] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:26.577374 14475 leader_election.cc:288] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:26.577636 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4d550eda29472d8c9ae494d25efc83" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:26.577775 9767 raft_consensus.cc:2454] T fe4d550eda29472d8c9ae494d25efc83 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:26.578085 6825 leader_election.cc:302] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:26.579617 14475 raft_consensus.cc:2781] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.579679 14475 raft_consensus.cc:482] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.579716 14475 raft_consensus.cc:3037] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.579838 6740 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet 8b9a601328d7447b9cdb75999ae153c9 (table rebalancer_dynamic_table [id=9b3249c1d06c4b73824b65c169bc9c95]). Expected version 1 got 0
I20240503 07:59:26.580936 14475 raft_consensus.cc:504] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:26.581329 14475 leader_election.cc:288] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:26.581615 6744 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 8b9a601328d7447b9cdb75999ae153c9 (table rebalancer_dynamic_table [id=9b3249c1d06c4b73824b65c169bc9c95]). Expected version 1 got 0
I20240503 07:59:26.583387 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4d550eda29472d8c9ae494d25efc83" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:26.583515 9767 raft_consensus.cc:3037] T fe4d550eda29472d8c9ae494d25efc83 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.584481 9767 raft_consensus.cc:2454] T fe4d550eda29472d8c9ae494d25efc83 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:26.585891 6825 leader_election.cc:302] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:26.586416 14629 raft_consensus.cc:2781] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.586513 14629 raft_consensus.cc:686] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:26.586509 15025 tablet.cc:1722] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.586645 14629 consensus_queue.cc:237] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:26.586884 15025 tablet.cc:1620] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:26.587297 15026 tablet.cc:1722] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.587616 15026 tablet.cc:1620] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:26.590065 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4d550eda29472d8c9ae494d25efc83" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:26.590215 9357 raft_consensus.cc:3037] T fe4d550eda29472d8c9ae494d25efc83 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.590932 6744 catalog_manager.cc:5484] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:26.591316 9357 raft_consensus.cc:2454] T fe4d550eda29472d8c9ae494d25efc83 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:26.592094 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4d550eda29472d8c9ae494d25efc83" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:26.592235 9374 raft_consensus.cc:2362] T fe4d550eda29472d8c9ae494d25efc83 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1. Re-sending same reply.
I20240503 07:59:26.593745 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=9b3249c1d06c4b73824b65c169bc9c95] alter complete (version 1)
I20240503 07:59:26.595268 15027 tablet.cc:1722] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.595716 15027 tablet.cc:1620] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:26.599727 14635 raft_consensus.cc:482] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.599843 14635 raft_consensus.cc:504] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.600268 14635 leader_election.cc:288] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:26.600977 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe38557414dd4f509604884bb23cc8ee" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:26.601099 9636 raft_consensus.cc:2454] T fe38557414dd4f509604884bb23cc8ee P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:26.601454 9700 leader_election.cc:302] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:26.601610 14635 raft_consensus.cc:2781] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.601662 14635 raft_consensus.cc:482] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.601692 14635 raft_consensus.cc:3037] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.602118 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe38557414dd4f509604884bb23cc8ee" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:26.602241 9374 raft_consensus.cc:2454] T fe38557414dd4f509604884bb23cc8ee P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:26.602834 14635 raft_consensus.cc:504] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.603166 14635 leader_election.cc:288] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:26.603806 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe38557414dd4f509604884bb23cc8ee" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:26.603935 9636 raft_consensus.cc:3037] T fe38557414dd4f509604884bb23cc8ee P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.604789 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe38557414dd4f509604884bb23cc8ee" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:26.604903 9374 raft_consensus.cc:3037] T fe38557414dd4f509604884bb23cc8ee P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.605053 9636 raft_consensus.cc:2454] T fe38557414dd4f509604884bb23cc8ee P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:26.605787 9374 raft_consensus.cc:2454] T fe38557414dd4f509604884bb23cc8ee P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:26.606676 9700 leader_election.cc:302] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:26.607550 14635 raft_consensus.cc:2781] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.607637 14635 raft_consensus.cc:686] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:26.607774 14635 consensus_queue.cc:237] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.610496 6744 catalog_manager.cc:5484] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:26.618152 15041 ts_tablet_manager.cc:902] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:26.618729 15041 tablet_copy_client.cc:308] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:26.619607 6908 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet cb2190946fb04a6392097f9588ffda6b from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:59233)
I20240503 07:59:26.619719 6908 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet cb2190946fb04a6392097f9588ffda6b from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:59233: session id = 159177b3caeb434baa98e04136d050b2-cb2190946fb04a6392097f9588ffda6b
I20240503 07:59:26.620644 6908 tablet_copy_source_session.cc:205] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:26.621909 15041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2190946fb04a6392097f9588ffda6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.624331 15041 tablet_copy_client.cc:783] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:26.624603 15041 tablet_copy_client.cc:647] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:26.626513 15041 tablet_copy_client.cc:523] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:26.628342 15041 tablet_bootstrap.cc:492] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:26.638908 9636 consensus_queue.cc:237] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:26.640566 9504 raft_consensus.cc:1261] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:26.640817 9569 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Couldn't send request to peer c4218904dd7e42d2be95b901a6e5db92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27139acdbdb54c248f35fcf9ec023f4d. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.641067 14820 consensus_queue.cc:1035] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.641961 9374 raft_consensus.cc:1261] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.642438 13406 consensus_queue.cc:1035] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.647102 15055 ts_tablet_manager.cc:902] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:26.647650 15055 tablet_copy_client.cc:308] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:26.648204 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet 50169f7d514d4fe3a4f846faaed03325 from peer 00a3f8e159c64327832d79fb32cdcca8 ({username='slave'} at 127.0.99.131:38079)
I20240503 07:59:26.648327 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet 50169f7d514d4fe3a4f846faaed03325 from peer 00a3f8e159c64327832d79fb32cdcca8 at {username='slave'} at 127.0.99.131:38079: session id = 00a3f8e159c64327832d79fb32cdcca8-50169f7d514d4fe3a4f846faaed03325
I20240503 07:59:26.649441 7302 tablet_copy_source_session.cc:205] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:26.650566 15055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50169f7d514d4fe3a4f846faaed03325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.651255 14629 raft_consensus.cc:482] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.651351 14629 raft_consensus.cc:504] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:26.651690 14629 leader_election.cc:288] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:26.652271 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae94f901c2d48968b87e091c2e085e4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:26.652397 9634 raft_consensus.cc:2454] T 3ae94f901c2d48968b87e091c2e085e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:26.653096 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae94f901c2d48968b87e091c2e085e4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:26.653239 7278 raft_consensus.cc:2454] T 3ae94f901c2d48968b87e091c2e085e4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:26.653565 6825 leader_election.cc:302] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:26.653749 15041 tablet_bootstrap.cc:492] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:26.653784 14475 raft_consensus.cc:2781] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.653837 14475 raft_consensus.cc:482] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.653872 14475 raft_consensus.cc:3037] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.654083 15041 tablet_bootstrap.cc:492] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:26.654109 15055 tablet_copy_client.cc:783] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:26.654263 15055 tablet_copy_client.cc:647] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:26.654282 15041 ts_tablet_manager.cc:1367] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.000s
I20240503 07:59:26.654760 15041 raft_consensus.cc:348] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.654866 15041 raft_consensus.cc:729] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.655021 15041 consensus_queue.cc:260] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.655166 14475 raft_consensus.cc:504] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:26.655313 15041 ts_tablet_manager.cc:1398] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 07:59:26.655558 14475 leader_election.cc:288] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:26.655532 13406 raft_consensus.cc:2932] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.656005 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae94f901c2d48968b87e091c2e085e4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:26.656121 9634 raft_consensus.cc:3037] T 3ae94f901c2d48968b87e091c2e085e4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.656992 9634 raft_consensus.cc:2454] T 3ae94f901c2d48968b87e091c2e085e4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:26.657773 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-cb2190946fb04a6392097f9588ffda6b received from {username='slave'} at 127.0.99.135:59233
I20240503 07:59:26.657866 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session 159177b3caeb434baa98e04136d050b2-cb2190946fb04a6392097f9588ffda6b on tablet cb2190946fb04a6392097f9588ffda6b with peer 159177b3caeb434baa98e04136d050b2
I20240503 07:59:26.658368 15055 tablet_copy_client.cc:523] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:26.659339 9374 raft_consensus.cc:2932] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.660120 15055 tablet_bootstrap.cc:492] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:26.661620 6744 catalog_manager.cc:5484] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.663650 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae94f901c2d48968b87e091c2e085e4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:26.663797 7278 raft_consensus.cc:3037] T 3ae94f901c2d48968b87e091c2e085e4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.664878 7278 raft_consensus.cc:2454] T 3ae94f901c2d48968b87e091c2e085e4 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:26.666278 15055 tablet_bootstrap.cc:492] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:26.666529 15055 tablet_bootstrap.cc:492] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap complete.
I20240503 07:59:26.666676 15055 ts_tablet_manager.cc:1367] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 07:59:26.657786 6822 leader_election.cc:302] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:26.667109 14475 raft_consensus.cc:2781] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.667191 14475 raft_consensus.cc:686] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:26.667191 15055 raft_consensus.cc:348] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:26.667320 15055 raft_consensus.cc:729] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.667356 14475 consensus_queue.cc:237] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:26.667593 15055 consensus_queue.cc:260] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:26.667932 15055 ts_tablet_manager.cc:1398] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.668588 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session 00a3f8e159c64327832d79fb32cdcca8-50169f7d514d4fe3a4f846faaed03325 received from {username='slave'} at 127.0.99.131:38079
I20240503 07:59:26.668659 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session 00a3f8e159c64327832d79fb32cdcca8-50169f7d514d4fe3a4f846faaed03325 on tablet 50169f7d514d4fe3a4f846faaed03325 with peer 00a3f8e159c64327832d79fb32cdcca8
I20240503 07:59:26.669724 6742 catalog_manager.cc:5484] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:26.671797 9504 raft_consensus.cc:2932] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.695806 9504 consensus_queue.cc:237] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:26.697412 6743 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:26.697631 9767 raft_consensus.cc:1261] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.698190 9634 raft_consensus.cc:1261] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:26.698554 9438 consensus_peers.cc:485] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a -> Peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Couldn't send request to peer c4218904dd7e42d2be95b901a6e5db92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8c22545fc9f4b248c2fdc568967f47f. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.698745 14554 consensus_queue.cc:1035] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.699105 15080 consensus_queue.cc:1035] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.701786 14554 raft_consensus.cc:2932] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.703173 9767 raft_consensus.cc:2932] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.703480 15025 tablet.cc:1722] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.703785 15025 tablet.cc:1620] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:26.704342 9634 raft_consensus.cc:2932] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.706604 15027 tablet.cc:1722] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.706715 6740 catalog_manager.cc:5484] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.706943 15027 tablet.cc:1620] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:26.707568 15026 tablet.cc:1722] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.707895 15026 tablet.cc:1620] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:26.715706 6744 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=9b3249c1d06c4b73824b65c169bc9c95] alter complete (version 2)
I20240503 07:59:26.732403 15100 ts_tablet_manager.cc:902] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:26.733376 15100 tablet_copy_client.cc:308] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:26.734251 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet 4737c826ebb64177a718d83b939c3e7b from peer 9c085a3241f14cb7ba1332b508f4b57d ({username='slave'} at 127.0.99.130:42749)
I20240503 07:59:26.734385 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet 4737c826ebb64177a718d83b939c3e7b from peer 9c085a3241f14cb7ba1332b508f4b57d at {username='slave'} at 127.0.99.130:42749: session id = 9c085a3241f14cb7ba1332b508f4b57d-4737c826ebb64177a718d83b939c3e7b
I20240503 07:59:26.734802 7278 consensus_queue.cc:237] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:26.735458 7302 tablet_copy_source_session.cc:205] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:26.736652 6886 raft_consensus.cc:1261] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:26.737126 14440 consensus_queue.cc:1035] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:26.737283 7149 raft_consensus.cc:1261] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:26.737527 15100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4737c826ebb64177a718d83b939c3e7b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 07:59:26.738273 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e63db30c0454722b6a794a9259a67c0. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.739149 14440 consensus_queue.cc:1035] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:26.740423 15100 tablet_copy_client.cc:783] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:26.740638 15100 tablet_copy_client.cc:647] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:26.741683 15100 tablet_copy_client.cc:523] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:26.743069 15100 tablet_bootstrap.cc:492] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:26.744055 14888 raft_consensus.cc:2932] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:26.744549 6886 raft_consensus.cc:2932] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:26.746048 7149 raft_consensus.cc:2932] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:26.747491 15100 tablet_bootstrap.cc:492] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:26.747725 15100 tablet_bootstrap.cc:492] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap complete.
I20240503 07:59:26.747889 15100 ts_tablet_manager.cc:1367] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s
I20240503 07:59:26.748028 6743 catalog_manager.cc:5484] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:26.748349 15100 raft_consensus.cc:348] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:26.748453 15100 raft_consensus.cc:729] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.748585 15100 consensus_queue.cc:260] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:26.748839 15041 ts_tablet_manager.cc:902] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:26.749285 15041 tablet_copy_client.cc:308] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:26.750319 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet 0fcc4c2eeaec4b359108ec64a6e20da8 from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:43259)
I20240503 07:59:26.750448 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet 0fcc4c2eeaec4b359108ec64a6e20da8 from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:43259: session id = 159177b3caeb434baa98e04136d050b2-0fcc4c2eeaec4b359108ec64a6e20da8
I20240503 07:59:26.751433 7302 tablet_copy_source_session.cc:205] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:26.751482 15100 ts_tablet_manager.cc:1398] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:26.752067 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-4737c826ebb64177a718d83b939c3e7b received from {username='slave'} at 127.0.99.130:42749
I20240503 07:59:26.752136 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-4737c826ebb64177a718d83b939c3e7b on tablet 4737c826ebb64177a718d83b939c3e7b with peer 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:26.752816 15041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcc4c2eeaec4b359108ec64a6e20da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.755033 15041 tablet_copy_client.cc:783] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:26.755179 15041 tablet_copy_client.cc:647] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:26.756353 15041 tablet_copy_client.cc:523] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:26.757943 15041 tablet_bootstrap.cc:492] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:26.774751 7278 consensus_queue.cc:237] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:26.776295 7019 raft_consensus.cc:1261] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:26.776731 7149 raft_consensus.cc:1261] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:26.776698 14888 consensus_queue.cc:1035] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:26.776995 7218 consensus_peers.cc:485] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc -> Peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Couldn't send request to peer a372e98a670f49e9adca46ab5c836c4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba1de17d15b74c4497e649aadc8569d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.777478 14888 consensus_queue.cc:1035] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:26.782063 15041 tablet_bootstrap.cc:492] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:26.782410 15041 tablet_bootstrap.cc:492] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:26.782608 15041 ts_tablet_manager.cc:1367] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.025s user 0.021s sys 0.003s
I20240503 07:59:26.782821 14888 raft_consensus.cc:2932] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:26.783136 15041 raft_consensus.cc:348] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.783253 15041 raft_consensus.cc:729] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.783386 15041 consensus_queue.cc:260] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:26.783674 15041 ts_tablet_manager.cc:1398] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.784929 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-0fcc4c2eeaec4b359108ec64a6e20da8 received from {username='slave'} at 127.0.99.135:43259
I20240503 07:59:26.785004 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session 159177b3caeb434baa98e04136d050b2-0fcc4c2eeaec4b359108ec64a6e20da8 on tablet 0fcc4c2eeaec4b359108ec64a6e20da8 with peer 159177b3caeb434baa98e04136d050b2
I20240503 07:59:26.785456 7149 raft_consensus.cc:2932] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:26.787117 6745 catalog_manager.cc:5484] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:26.787163 7019 raft_consensus.cc:2932] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:26.808077 9374 consensus_queue.cc:237] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:26.809636 9504 raft_consensus.cc:1261] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.809876 7149 raft_consensus.cc:1261] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.810604 15151 consensus_queue.cc:1035] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:26.810822 14655 consensus_queue.cc:1035] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:26.810974 9307 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b -> Peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Couldn't send request to peer c4218904dd7e42d2be95b901a6e5db92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21f55eb986bd4b7bbda10e683f3bfcad. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.813251 14655 raft_consensus.cc:2932] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.813753 9504 raft_consensus.cc:2932] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.815860 7149 raft_consensus.cc:2932] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.816879 6744 catalog_manager.cc:5484] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:26.818789 6743 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:26.826408 15025 tablet.cc:1722] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.826776 15026 tablet.cc:1722] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.831645 15027 tablet.cc:1722] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.832371 6743 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=9b3249c1d06c4b73824b65c169bc9c95] alter complete (version 3)
I20240503 07:59:26.833256 15164 ts_tablet_manager.cc:902] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:26.834079 15164 tablet_copy_client.cc:308] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:26.835806 6908 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet cf8e0a0da8b941e28e797de8f8b7a3f4 from peer a372e98a670f49e9adca46ab5c836c4a ({username='slave'} at 127.0.99.136:39953)
I20240503 07:59:26.835924 6908 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet cf8e0a0da8b941e28e797de8f8b7a3f4 from peer a372e98a670f49e9adca46ab5c836c4a at {username='slave'} at 127.0.99.136:39953: session id = a372e98a670f49e9adca46ab5c836c4a-cf8e0a0da8b941e28e797de8f8b7a3f4
I20240503 07:59:26.836911 6908 tablet_copy_source_session.cc:205] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:26.838059 15164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf8e0a0da8b941e28e797de8f8b7a3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.840385 15164 tablet_copy_client.cc:783] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:26.840555 15164 tablet_copy_client.cc:647] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:26.842242 15164 tablet_copy_client.cc:523] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:26.843430 15164 tablet_bootstrap.cc:492] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:26.846870 15164 tablet_bootstrap.cc:492] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:26.847088 15164 tablet_bootstrap.cc:492] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap complete.
I20240503 07:59:26.847244 15164 ts_tablet_manager.cc:1367] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240503 07:59:26.847678 15164 raft_consensus.cc:348] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:26.847787 15164 raft_consensus.cc:729] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.847913 15164 consensus_queue.cc:260] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:26.848242 15164 ts_tablet_manager.cc:1398] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:26.848779 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session a372e98a670f49e9adca46ab5c836c4a-cf8e0a0da8b941e28e797de8f8b7a3f4 received from {username='slave'} at 127.0.99.136:39953
I20240503 07:59:26.848863 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session a372e98a670f49e9adca46ab5c836c4a-cf8e0a0da8b941e28e797de8f8b7a3f4 on tablet cf8e0a0da8b941e28e797de8f8b7a3f4 with peer a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:26.864723 15181 ts_tablet_manager.cc:902] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:26.865289 15181 tablet_copy_client.cc:308] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:26.866307 6908 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet f365955974634bc58e8a0da0fc2563d7 from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:50775)
I20240503 07:59:26.866449 6908 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet f365955974634bc58e8a0da0fc2563d7 from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:50775: session id = cb73994515af4880b05383e78368bb4a-f365955974634bc58e8a0da0fc2563d7
I20240503 07:59:26.867583 6908 tablet_copy_source_session.cc:205] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:26.869660 15181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f365955974634bc58e8a0da0fc2563d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.873214 15181 tablet_copy_client.cc:783] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:26.873479 15181 tablet_copy_client.cc:647] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:26.874723 15181 tablet_copy_client.cc:523] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:26.876272 15181 tablet_bootstrap.cc:492] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:26.905483 15181 tablet_bootstrap.cc:492] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:26.906069 15181 tablet_bootstrap.cc:492] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:26.906380 15181 ts_tablet_manager.cc:1367] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.000s
I20240503 07:59:26.906942 15181 raft_consensus.cc:348] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:26.907090 15181 raft_consensus.cc:729] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.907229 15181 consensus_queue.cc:260] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:26.907653 15181 ts_tablet_manager.cc:1398] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.908167 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-f365955974634bc58e8a0da0fc2563d7 received from {username='slave'} at 127.0.99.134:50775
I20240503 07:59:26.908262 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session cb73994515af4880b05383e78368bb4a-f365955974634bc58e8a0da0fc2563d7 on tablet f365955974634bc58e8a0da0fc2563d7 with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:26.932895 7129 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:26.936111 9485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:26.942049 6742 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:26.942246 6742 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:26.942600 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:26.944964 6742 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 8b9a601328d7447b9cdb75999ae153c9
I20240503 07:59:26.945744 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9a601328d7447b9cdb75999ae153c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:26 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:26.946053 15218 tablet_replica.cc:330] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:26.946197 15218 raft_consensus.cc:2229] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:26.946229 9745 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9a601328d7447b9cdb75999ae153c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:26 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:26.946408 15218 raft_consensus.cc:2258] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.946887 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:26.947279 15218 ts_tablet_manager.cc:1875] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:26.948797 9351 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9a601328d7447b9cdb75999ae153c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:26 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:26.950695 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:26.951511 15220 tablet_replica.cc:330] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:26.951700 15220 raft_consensus.cc:2229] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:26.951879 9615 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:26.951905 15220 raft_consensus.cc:2258] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.952219 6867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:26.954195 15219 tablet_replica.cc:330] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:26.954434 15219 raft_consensus.cc:2229] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:26.954699 15219 raft_consensus.cc:2258] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.957201 15219 ts_tablet_manager.cc:1875] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:26.958889 15220 ts_tablet_manager.cc:1875] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:26.958853 15218 ts_tablet_manager.cc:1888] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:26.959045 15218 log.cc:1196] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/8b9a601328d7447b9cdb75999ae153c9
I20240503 07:59:26.959506 15218 ts_tablet_manager.cc:1909] T 8b9a601328d7447b9cdb75999ae153c9 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:26.961189 15220 ts_tablet_manager.cc:1888] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:26.961269 15220 log.cc:1196] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/8b9a601328d7447b9cdb75999ae153c9
I20240503 07:59:26.961519 15220 ts_tablet_manager.cc:1909] T 8b9a601328d7447b9cdb75999ae153c9 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:26.962694 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 8b9a601328d7447b9cdb75999ae153c9 (table rebalancer_dynamic_table_new_name [id=9b3249c1d06c4b73824b65c169bc9c95]) successfully deleted
I20240503 07:59:26.963383 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 8b9a601328d7447b9cdb75999ae153c9 (table rebalancer_dynamic_table_new_name [id=9b3249c1d06c4b73824b65c169bc9c95]) successfully deleted
I20240503 07:59:26.974743 9351 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:26.977285 7278 raft_consensus.cc:1203] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:26.993579 15219 ts_tablet_manager.cc:1888] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:26.993692 15219 log.cc:1196] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/8b9a601328d7447b9cdb75999ae153c9
I20240503 07:59:26.993970 15219 ts_tablet_manager.cc:1909] T 8b9a601328d7447b9cdb75999ae153c9 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:27.006424 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 8b9a601328d7447b9cdb75999ae153c9 (table rebalancer_dynamic_table_new_name [id=9b3249c1d06c4b73824b65c169bc9c95]) successfully deleted
I20240503 07:59:27.024125 9634 raft_consensus.cc:1203] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:27.046437 14635 consensus_queue.cc:1035] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.047793 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:27.052484 15083 consensus_queue.cc:1035] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.059353 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 2441f7128305466fbcb0ef64226f34e9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5bde7b0605e545aab55481f70236069b]), partition=
I20240503 07:59:27.059353 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 2441f7128305466fbcb0ef64226f34e9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5bde7b0605e545aab55481f70236069b]), partition=
I20240503 07:59:27.059646 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2441f7128305466fbcb0ef64226f34e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.059651 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2441f7128305466fbcb0ef64226f34e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.062543 15236 tablet_bootstrap.cc:492] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:27.062857 9351 tablet_service.cc:1467] Processing CreateTablet for tablet 2441f7128305466fbcb0ef64226f34e9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5bde7b0605e545aab55481f70236069b]), partition=
I20240503 07:59:27.063120 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2441f7128305466fbcb0ef64226f34e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.063804 15236 tablet_bootstrap.cc:654] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.064538 15235 tablet_bootstrap.cc:492] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:27.065635 15236 tablet_bootstrap.cc:492] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:27.065740 15236 ts_tablet_manager.cc:1367] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:27.065920 15235 tablet_bootstrap.cc:654] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.066287 15236 raft_consensus.cc:348] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:27.066776 15238 tablet_bootstrap.cc:492] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:27.067703 15238 tablet_bootstrap.cc:654] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.067847 15236 raft_consensus.cc:374] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.067905 15236 raft_consensus.cc:729] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.068034 15236 consensus_queue.cc:260] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:27.068334 15236 ts_tablet_manager.cc:1398] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:27.069411 15235 tablet_bootstrap.cc:492] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:27.069507 15235 ts_tablet_manager.cc:1367] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:27.070003 15235 raft_consensus.cc:348] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:27.070130 15235 raft_consensus.cc:374] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.070214 15235 raft_consensus.cc:729] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.070399 15235 consensus_queue.cc:260] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:27.070767 15235 ts_tablet_manager.cc:1398] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.073128 6886 raft_consensus.cc:1203] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:27.077807 15238 tablet_bootstrap.cc:492] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:27.078027 15238 ts_tablet_manager.cc:1367] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 07:59:27.078635 15238 raft_consensus.cc:348] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:27.078768 15238 raft_consensus.cc:374] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.078835 15238 raft_consensus.cc:729] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.078969 15238 consensus_queue.cc:260] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:27.079913 15238 ts_tablet_manager.cc:1398] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:27.083740 15061 raft_consensus.cc:482] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.083860 15061 raft_consensus.cc:504] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:27.084255 15061 leader_election.cc:288] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:27.084502 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2441f7128305466fbcb0ef64226f34e9" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:27.084614 9374 raft_consensus.cc:2454] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:27.084720 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2441f7128305466fbcb0ef64226f34e9" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:27.084849 9767 raft_consensus.cc:2454] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:27.084893 9570 leader_election.cc:302] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:27.085047 15061 raft_consensus.cc:2781] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.085098 15061 raft_consensus.cc:482] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.085141 15061 raft_consensus.cc:3037] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.086045 15061 raft_consensus.cc:504] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:27.086418 15061 leader_election.cc:288] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:27.086721 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2441f7128305466fbcb0ef64226f34e9" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:27.086866 9374 raft_consensus.cc:3037] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.086977 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2441f7128305466fbcb0ef64226f34e9" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:27.087085 9767 raft_consensus.cc:3037] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.087931 9374 raft_consensus.cc:2454] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:27.087976 9767 raft_consensus.cc:2454] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:27.088182 9570 leader_election.cc:302] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:27.088511 15060 raft_consensus.cc:2781] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.088605 15060 raft_consensus.cc:686] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:27.088750 15060 consensus_queue.cc:237] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:27.091207 6742 catalog_manager.cc:5484] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:27.094185 14475 consensus_queue.cc:1035] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.097391 14475 consensus_queue.cc:1035] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.099220 15274 ts_tablet_manager.cc:902] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92: Initiating tablet copy from peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:27.099776 15274 tablet_copy_client.cc:308] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:43369
I20240503 07:59:27.100208 9656 tablet_copy_service.cc:140] P 159177b3caeb434baa98e04136d050b2: Received BeginTabletCopySession request for tablet 27139acdbdb54c248f35fcf9ec023f4d from peer c4218904dd7e42d2be95b901a6e5db92 ({username='slave'} at 127.0.99.129:40095)
I20240503 07:59:27.100312 9656 tablet_copy_service.cc:161] P 159177b3caeb434baa98e04136d050b2: Beginning new tablet copy session on tablet 27139acdbdb54c248f35fcf9ec023f4d from peer c4218904dd7e42d2be95b901a6e5db92 at {username='slave'} at 127.0.99.129:40095: session id = c4218904dd7e42d2be95b901a6e5db92-27139acdbdb54c248f35fcf9ec023f4d
I20240503 07:59:27.101310 9656 tablet_copy_source_session.cc:205] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:27.102403 15274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27139acdbdb54c248f35fcf9ec023f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.107667 15274 tablet_copy_client.cc:783] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:27.107901 15274 tablet_copy_client.cc:647] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:27.110944 15274 tablet_copy_client.cc:523] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:27.112919 15274 tablet_bootstrap.cc:492] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:27.116916 15274 tablet_bootstrap.cc:492] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:27.117174 15274 tablet_bootstrap.cc:492] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92: Bootstrap complete.
I20240503 07:59:27.117353 15274 ts_tablet_manager.cc:1367] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 07:59:27.117759 15274 raft_consensus.cc:348] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:27.117854 15274 raft_consensus.cc:729] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.117969 15274 consensus_queue.cc:260] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:27.118261 15274 ts_tablet_manager.cc:1398] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:27.118772 9656 tablet_copy_service.cc:342] P 159177b3caeb434baa98e04136d050b2: Request end of tablet copy session c4218904dd7e42d2be95b901a6e5db92-27139acdbdb54c248f35fcf9ec023f4d received from {username='slave'} at 127.0.99.129:40095
I20240503 07:59:27.118856 9656 tablet_copy_service.cc:433] P 159177b3caeb434baa98e04136d050b2: ending tablet copy session c4218904dd7e42d2be95b901a6e5db92-27139acdbdb54c248f35fcf9ec023f4d on tablet 27139acdbdb54c248f35fcf9ec023f4d with peer c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:27.121877 14475 consensus_queue.cc:1035] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.129092 14629 consensus_queue.cc:1035] T 3ae94f901c2d48968b87e091c2e085e4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.140482 7149 raft_consensus.cc:1203] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:27.172251 9351 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:27.172398 9745 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:27.173974 7129 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:27.174469 6867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:27.187137 9615 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:27.195874 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:27.196031 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:27.203924 9374 raft_consensus.cc:1261] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:27.203948 9767 raft_consensus.cc:1261] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:27.204419 15060 consensus_queue.cc:1035] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.204603 15061 consensus_queue.cc:1035] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.206503 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:27.208846 7129 tablet_service.cc:1467] Processing CreateTablet for tablet c2b5e8db1eec44a7881dde976e8a0a12 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=646fa21ff4b6475fbbcfb8bb19f75c68]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.208853 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 84ffde5aaa3b4a67a878c77ef73472d6 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=646fa21ff4b6475fbbcfb8bb19f75c68]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.209211 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b5e8db1eec44a7881dde976e8a0a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.209213 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ffde5aaa3b4a67a878c77ef73472d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.209625 9615 tablet_service.cc:1467] Processing CreateTablet for tablet c2b5e8db1eec44a7881dde976e8a0a12 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=646fa21ff4b6475fbbcfb8bb19f75c68]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.209903 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b5e8db1eec44a7881dde976e8a0a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.212088 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 84ffde5aaa3b4a67a878c77ef73472d6 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=646fa21ff4b6475fbbcfb8bb19f75c68]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.212236 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6388b40cba4095b15b1fdbb1f2e9f9 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=646fa21ff4b6475fbbcfb8bb19f75c68]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.212327 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ffde5aaa3b4a67a878c77ef73472d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.212510 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6388b40cba4095b15b1fdbb1f2e9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.212666 15236 tablet_bootstrap.cc:492] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:27.213799 15236 tablet_bootstrap.cc:654] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.214668 9351 tablet_service.cc:1467] Processing CreateTablet for tablet c2b5e8db1eec44a7881dde976e8a0a12 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=646fa21ff4b6475fbbcfb8bb19f75c68]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.214951 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b5e8db1eec44a7881dde976e8a0a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.215152 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6388b40cba4095b15b1fdbb1f2e9f9 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=646fa21ff4b6475fbbcfb8bb19f75c68]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.215415 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6388b40cba4095b15b1fdbb1f2e9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.216524 6740 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 2441f7128305466fbcb0ef64226f34e9 (table rebalancer_dynamic_table [id=5bde7b0605e545aab55481f70236069b]). Expected version 1 got 0
I20240503 07:59:27.217053 15235 tablet_bootstrap.cc:492] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:27.217394 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6388b40cba4095b15b1fdbb1f2e9f9 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=646fa21ff4b6475fbbcfb8bb19f75c68]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.217660 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6388b40cba4095b15b1fdbb1f2e9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.217809 15238 tablet_bootstrap.cc:492] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:27.218066 15235 tablet_bootstrap.cc:654] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.218783 15238 tablet_bootstrap.cc:654] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.219156 15299 tablet_bootstrap.cc:492] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:27.220129 15299 tablet_bootstrap.cc:654] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.220793 15238 tablet_bootstrap.cc:492] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:27.220887 15238 ts_tablet_manager.cc:1367] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:27.221336 15238 raft_consensus.cc:348] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.221455 15238 raft_consensus.cc:374] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.221482 15238 raft_consensus.cc:729] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.221575 15238 consensus_queue.cc:260] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.221853 15238 ts_tablet_manager.cc:1398] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:27.222312 6738 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet 2441f7128305466fbcb0ef64226f34e9 (table rebalancer_dynamic_table [id=5bde7b0605e545aab55481f70236069b]). Expected version 1 got 0
I20240503 07:59:27.223591 15235 tablet_bootstrap.cc:492] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:27.223681 15235 ts_tablet_manager.cc:1367] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:27.224138 15235 raft_consensus.cc:348] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.224236 15235 raft_consensus.cc:374] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.224261 15235 raft_consensus.cc:729] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.224391 15235 consensus_queue.cc:260] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.224718 15235 ts_tablet_manager.cc:1398] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.226992 9636 raft_consensus.cc:1203] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:27.228483 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 84ffde5aaa3b4a67a878c77ef73472d6 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=646fa21ff4b6475fbbcfb8bb19f75c68]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.228832 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ffde5aaa3b4a67a878c77ef73472d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.230365 15234 mvcc.cc:205] Tried to move back new op lower bound from 7023506092861435904 to 7023506092397076480. Current Snapshot: MvccSnapshot[applied={T|T < 7023506092861435904}]
I20240503 07:59:27.233996 15025 tablet.cc:1722] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:27.234419 15025 tablet.cc:1620] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:27.234773 15026 tablet.cc:1722] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:27.235105 15026 tablet.cc:1620] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:27.237898 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:27.243386 15307 tablet.cc:1722] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:27.243891 15307 tablet.cc:1620] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:27.246300 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=5bde7b0605e545aab55481f70236069b] alter complete (version 1)
I20240503 07:59:27.260462 9485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:27.265033 15299 tablet_bootstrap.cc:492] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:27.265183 15299 ts_tablet_manager.cc:1367] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20240503 07:59:27.265730 15299 raft_consensus.cc:348] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.265870 15299 raft_consensus.cc:374] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.265904 15299 raft_consensus.cc:729] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.266041 15299 consensus_queue.cc:260] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.266626 15299 ts_tablet_manager.cc:1398] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:27.266798 15299 tablet_bootstrap.cc:492] T 84ffde5aaa3b4a67a878c77ef73472d6 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:27.267850 15299 tablet_bootstrap.cc:654] T 84ffde5aaa3b4a67a878c77ef73472d6 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.269095 15274 ts_tablet_manager.cc:902] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92: Initiating tablet copy from peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:27.269512 15274 tablet_copy_client.cc:308] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:45435
I20240503 07:59:27.271586 9525 tablet_copy_service.cc:140] P cb73994515af4880b05383e78368bb4a: Received BeginTabletCopySession request for tablet a8c22545fc9f4b248c2fdc568967f47f from peer c4218904dd7e42d2be95b901a6e5db92 ({username='slave'} at 127.0.99.129:59589)
I20240503 07:59:27.271699 9525 tablet_copy_service.cc:161] P cb73994515af4880b05383e78368bb4a: Beginning new tablet copy session on tablet a8c22545fc9f4b248c2fdc568967f47f from peer c4218904dd7e42d2be95b901a6e5db92 at {username='slave'} at 127.0.99.129:59589: session id = c4218904dd7e42d2be95b901a6e5db92-a8c22545fc9f4b248c2fdc568967f47f
I20240503 07:59:27.272693 9525 tablet_copy_source_session.cc:205] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:27.273623 15274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c22545fc9f4b248c2fdc568967f47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.276304 15274 tablet_copy_client.cc:783] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:27.277869 15274 tablet_copy_client.cc:647] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:27.279896 15298 tablet_bootstrap.cc:492] T 84ffde5aaa3b4a67a878c77ef73472d6 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:27.282023 15164 ts_tablet_manager.cc:902] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:27.282526 15164 tablet_copy_client.cc:308] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:27.283104 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet ba1de17d15b74c4497e649aadc8569d2 from peer a372e98a670f49e9adca46ab5c836c4a ({username='slave'} at 127.0.99.136:41643)
I20240503 07:59:27.283217 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet ba1de17d15b74c4497e649aadc8569d2 from peer a372e98a670f49e9adca46ab5c836c4a at {username='slave'} at 127.0.99.136:41643: session id = a372e98a670f49e9adca46ab5c836c4a-ba1de17d15b74c4497e649aadc8569d2
I20240503 07:59:27.284358 7302 tablet_copy_source_session.cc:205] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:27.286556 15164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1de17d15b74c4497e649aadc8569d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.286798 15274 tablet_copy_client.cc:523] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:27.288317 15274 tablet_bootstrap.cc:492] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:27.288966 15164 tablet_copy_client.cc:783] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:27.289116 15164 tablet_copy_client.cc:647] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:27.290769 15299 tablet_bootstrap.cc:492] T 84ffde5aaa3b4a67a878c77ef73472d6 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:27.290876 15299 ts_tablet_manager.cc:1367] T 84ffde5aaa3b4a67a878c77ef73472d6 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20240503 07:59:27.291383 15299 raft_consensus.cc:348] T 84ffde5aaa3b4a67a878c77ef73472d6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.291505 15299 raft_consensus.cc:374] T 84ffde5aaa3b4a67a878c77ef73472d6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.291556 15299 raft_consensus.cc:729] T 84ffde5aaa3b4a67a878c77ef73472d6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.291692 15299 consensus_queue.cc:260] T 84ffde5aaa3b4a67a878c77ef73472d6 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.291999 15299 ts_tablet_manager.cc:1398] T 84ffde5aaa3b4a67a878c77ef73472d6 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:27.293531 15236 tablet_bootstrap.cc:492] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:27.293655 15236 ts_tablet_manager.cc:1367] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.000s
I20240503 07:59:27.294140 15236 raft_consensus.cc:348] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.294253 15236 raft_consensus.cc:374] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.294278 15236 raft_consensus.cc:729] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.294417 15236 consensus_queue.cc:260] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.294708 15236 ts_tablet_manager.cc:1398] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:27.294883 15236 tablet_bootstrap.cc:492] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:27.295818 15236 tablet_bootstrap.cc:654] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.299050 15164 tablet_copy_client.cc:523] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:27.300218 15298 tablet_bootstrap.cc:654] T 84ffde5aaa3b4a67a878c77ef73472d6 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.300695 15164 tablet_bootstrap.cc:492] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:27.302443 15236 tablet_bootstrap.cc:492] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:27.302538 15236 ts_tablet_manager.cc:1367] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 07:59:27.303005 15236 raft_consensus.cc:348] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.303117 15236 raft_consensus.cc:374] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.303148 15236 raft_consensus.cc:729] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.303265 15236 consensus_queue.cc:260] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.303542 15236 ts_tablet_manager.cc:1398] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.303719 15236 tablet_bootstrap.cc:492] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:27.304620 15236 tablet_bootstrap.cc:654] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.306196 15317 ts_tablet_manager.cc:902] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:27.306813 15317 tablet_copy_client.cc:308] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:27.307312 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet 6e63db30c0454722b6a794a9259a67c0 from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:56405)
I20240503 07:59:27.307433 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet 6e63db30c0454722b6a794a9259a67c0 from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:56405: session id = 3befbb2d206f4be89d3e07f38837374b-6e63db30c0454722b6a794a9259a67c0
I20240503 07:59:27.308223 15298 tablet_bootstrap.cc:492] T 84ffde5aaa3b4a67a878c77ef73472d6 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:27.308310 15298 ts_tablet_manager.cc:1367] T 84ffde5aaa3b4a67a878c77ef73472d6 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240503 07:59:27.308490 7302 tablet_copy_source_session.cc:205] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:27.308786 15298 raft_consensus.cc:348] T 84ffde5aaa3b4a67a878c77ef73472d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.308919 15298 raft_consensus.cc:374] T 84ffde5aaa3b4a67a878c77ef73472d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.308952 15298 raft_consensus.cc:729] T 84ffde5aaa3b4a67a878c77ef73472d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.309111 15298 consensus_queue.cc:260] T 84ffde5aaa3b4a67a878c77ef73472d6 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.309476 15298 ts_tablet_manager.cc:1398] T 84ffde5aaa3b4a67a878c77ef73472d6 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.309942 15298 tablet_bootstrap.cc:492] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:27.310879 15298 tablet_bootstrap.cc:654] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.311378 15317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e63db30c0454722b6a794a9259a67c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.311837 15236 tablet_bootstrap.cc:492] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:27.311939 15236 ts_tablet_manager.cc:1367] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:27.312347 15236 raft_consensus.cc:348] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.312458 15236 raft_consensus.cc:374] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.312489 15236 raft_consensus.cc:729] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.312606 15236 consensus_queue.cc:260] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.312877 15236 ts_tablet_manager.cc:1398] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.314245 15298 tablet_bootstrap.cc:492] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:27.314530 15298 ts_tablet_manager.cc:1367] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:27.315063 15298 raft_consensus.cc:348] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.315163 15298 raft_consensus.cc:374] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.315193 15298 raft_consensus.cc:729] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.315330 15298 consensus_queue.cc:260] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.315518 15274 tablet_bootstrap.cc:492] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:27.315699 15298 ts_tablet_manager.cc:1398] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.315799 15274 tablet_bootstrap.cc:492] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92: Bootstrap complete.
I20240503 07:59:27.315977 15274 ts_tablet_manager.cc:1367] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240503 07:59:27.316476 15274 raft_consensus.cc:348] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:27.316597 15274 raft_consensus.cc:729] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.316669 15317 tablet_copy_client.cc:783] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:27.316753 15274 consensus_queue.cc:260] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:27.316825 15317 tablet_copy_client.cc:647] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:27.317054 15274 ts_tablet_manager.cc:1398] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:27.319764 9525 tablet_copy_service.cc:342] P cb73994515af4880b05383e78368bb4a: Request end of tablet copy session c4218904dd7e42d2be95b901a6e5db92-a8c22545fc9f4b248c2fdc568967f47f received from {username='slave'} at 127.0.99.129:59589
I20240503 07:59:27.319854 9525 tablet_copy_service.cc:433] P cb73994515af4880b05383e78368bb4a: ending tablet copy session c4218904dd7e42d2be95b901a6e5db92-a8c22545fc9f4b248c2fdc568967f47f on tablet a8c22545fc9f4b248c2fdc568967f47f with peer c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:27.324672 15317 tablet_copy_client.cc:523] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:27.326638 15317 tablet_bootstrap.cc:492] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:27.336189 15083 raft_consensus.cc:482] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.336318 15083 raft_consensus.cc:504] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.336776 15083 leader_election.cc:288] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:27.337060 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6388b40cba4095b15b1fdbb1f2e9f9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:27.337214 7019 raft_consensus.cc:2454] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:27.337281 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6388b40cba4095b15b1fdbb1f2e9f9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:27.337435 9636 raft_consensus.cc:2454] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:27.337522 9702 leader_election.cc:302] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:27.338037 15083 raft_consensus.cc:2781] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.338095 15083 raft_consensus.cc:482] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.338145 15083 raft_consensus.cc:3037] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.338683 9767 raft_consensus.cc:1203] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:27.338761 9504 raft_consensus.cc:1203] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:27.339330 15083 raft_consensus.cc:504] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.339708 15083 leader_election.cc:288] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:27.340227 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6388b40cba4095b15b1fdbb1f2e9f9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:27.340356 7019 raft_consensus.cc:3037] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.340521 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6388b40cba4095b15b1fdbb1f2e9f9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:27.340642 9636 raft_consensus.cc:3037] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.341450 7019 raft_consensus.cc:2454] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:27.341816 9636 raft_consensus.cc:2454] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:27.341981 9702 leader_election.cc:302] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:27.342494 15083 raft_consensus.cc:2781] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.342586 15083 raft_consensus.cc:686] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:27.342744 15083 consensus_queue.cc:237] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.343907 7019 raft_consensus.cc:1203] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:27.345707 6740 catalog_manager.cc:5484] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:27.354506 15164 tablet_bootstrap.cc:492] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:27.354996 15164 tablet_bootstrap.cc:492] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap complete.
I20240503 07:59:27.355244 15164 ts_tablet_manager.cc:1367] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.055s user 0.023s sys 0.003s
I20240503 07:59:27.355800 15164 raft_consensus.cc:348] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:27.355912 15164 raft_consensus.cc:729] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.356048 15164 consensus_queue.cc:260] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:27.356539 15164 ts_tablet_manager.cc:1398] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.357151 15317 tablet_bootstrap.cc:492] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:27.357249 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session a372e98a670f49e9adca46ab5c836c4a-ba1de17d15b74c4497e649aadc8569d2 received from {username='slave'} at 127.0.99.136:41643
I20240503 07:59:27.357331 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session a372e98a670f49e9adca46ab5c836c4a-ba1de17d15b74c4497e649aadc8569d2 on tablet ba1de17d15b74c4497e649aadc8569d2 with peer a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:27.357470 15317 tablet_bootstrap.cc:492] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:27.357774 15317 ts_tablet_manager.cc:1367] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.000s
I20240503 07:59:27.358228 15317 raft_consensus.cc:348] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:27.358357 15317 raft_consensus.cc:729] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.358520 15317 consensus_queue.cc:260] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:27.358816 15317 ts_tablet_manager.cc:1398] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:27.359750 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-6e63db30c0454722b6a794a9259a67c0 received from {username='slave'} at 127.0.99.133:56405
I20240503 07:59:27.359825 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-6e63db30c0454722b6a794a9259a67c0 on tablet 6e63db30c0454722b6a794a9259a67c0 with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:27.365754 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:27.371860 15307 tablet.cc:1722] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:27.372256 15307 tablet.cc:1620] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:27.372749 15025 tablet.cc:1722] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:27.373034 15025 tablet.cc:1620] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:27.373064 15026 tablet.cc:1722] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:27.373342 15026 tablet.cc:1620] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:27.376567 15274 ts_tablet_manager.cc:902] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92: Initiating tablet copy from peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:27.377050 15274 tablet_copy_client.cc:308] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37815
I20240503 07:59:27.377509 9394 tablet_copy_service.cc:140] P 3befbb2d206f4be89d3e07f38837374b: Received BeginTabletCopySession request for tablet 21f55eb986bd4b7bbda10e683f3bfcad from peer c4218904dd7e42d2be95b901a6e5db92 ({username='slave'} at 127.0.99.129:37247)
I20240503 07:59:27.377609 9394 tablet_copy_service.cc:161] P 3befbb2d206f4be89d3e07f38837374b: Beginning new tablet copy session on tablet 21f55eb986bd4b7bbda10e683f3bfcad from peer c4218904dd7e42d2be95b901a6e5db92 at {username='slave'} at 127.0.99.129:37247: session id = c4218904dd7e42d2be95b901a6e5db92-21f55eb986bd4b7bbda10e683f3bfcad
I20240503 07:59:27.378466 9394 tablet_copy_source_session.cc:205] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:27.379400 15274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21f55eb986bd4b7bbda10e683f3bfcad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.381794 15274 tablet_copy_client.cc:783] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:27.381955 15274 tablet_copy_client.cc:647] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:27.386853 15274 tablet_copy_client.cc:523] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:27.387254 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=5bde7b0605e545aab55481f70236069b] alter complete (version 2)
I20240503 07:59:27.389045 15274 tablet_bootstrap.cc:492] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:27.393358 15274 tablet_bootstrap.cc:492] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:27.393621 15274 tablet_bootstrap.cc:492] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92: Bootstrap complete.
I20240503 07:59:27.393774 15274 ts_tablet_manager.cc:1367] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:27.394281 15274 raft_consensus.cc:348] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:27.394447 15274 raft_consensus.cc:729] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.394574 15274 consensus_queue.cc:260] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:27.394867 15274 ts_tablet_manager.cc:1398] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:27.395812 9394 tablet_copy_service.cc:342] P 3befbb2d206f4be89d3e07f38837374b: Request end of tablet copy session c4218904dd7e42d2be95b901a6e5db92-21f55eb986bd4b7bbda10e683f3bfcad received from {username='slave'} at 127.0.99.129:37247
I20240503 07:59:27.395905 9394 tablet_copy_service.cc:433] P 3befbb2d206f4be89d3e07f38837374b: ending tablet copy session c4218904dd7e42d2be95b901a6e5db92-21f55eb986bd4b7bbda10e683f3bfcad on tablet 21f55eb986bd4b7bbda10e683f3bfcad with peer c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:27.402171 13216 raft_consensus.cc:482] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.402484 13216 raft_consensus.cc:504] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.402915 13216 leader_election.cc:288] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:27.403358 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5e8db1eec44a7881dde976e8a0a12" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:27.403358 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5e8db1eec44a7881dde976e8a0a12" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:27.403497 7149 raft_consensus.cc:2454] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:27.403497 9636 raft_consensus.cc:2454] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:27.403829 9310 leader_election.cc:302] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:27.404145 13216 raft_consensus.cc:2781] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.404203 13216 raft_consensus.cc:482] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.404238 13216 raft_consensus.cc:3037] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.405251 13216 raft_consensus.cc:504] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.405640 13216 leader_election.cc:288] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:27.406260 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5e8db1eec44a7881dde976e8a0a12" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:27.406260 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5e8db1eec44a7881dde976e8a0a12" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:27.406402 7149 raft_consensus.cc:3037] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.406402 9636 raft_consensus.cc:3037] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.407433 9636 raft_consensus.cc:2454] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:27.407433 7149 raft_consensus.cc:2454] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:27.407904 9310 leader_election.cc:302] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:27.408082 13216 raft_consensus.cc:2781] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.408170 13216 raft_consensus.cc:686] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:27.408353 13216 consensus_queue.cc:237] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.410977 6746 catalog_manager.cc:5484] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:27.483723 6886 raft_consensus.cc:1203] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:27.489231 6742 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:27.494820 15307 tablet.cc:1722] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:27.495764 15025 tablet.cc:1722] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:27.499114 15026 tablet.cc:1722] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:27.501621 6742 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=5bde7b0605e545aab55481f70236069b] alter complete (version 3)
I20240503 07:59:27.602979 6745 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:27.603133 6745 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:27.605387 6745 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 2441f7128305466fbcb0ef64226f34e9
I20240503 07:59:27.605968 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet 2441f7128305466fbcb0ef64226f34e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:27.606055 9745 tablet_service.cc:1514] Processing DeleteTablet for tablet 2441f7128305466fbcb0ef64226f34e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:27.606400 15378 tablet_replica.cc:330] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:27.606412 15377 tablet_replica.cc:330] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:27.606564 15377 raft_consensus.cc:2229] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:27.606571 15378 raft_consensus.cc:2229] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:27.606701 15378 raft_consensus.cc:2258] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.606909 15377 raft_consensus.cc:2258] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.606935 9351 tablet_service.cc:1514] Processing DeleteTablet for tablet 2441f7128305466fbcb0ef64226f34e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:27.607352 15379 tablet_replica.cc:330] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:27.607395 15378 ts_tablet_manager.cc:1875] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.607517 15377 ts_tablet_manager.cc:1875] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.607587 15379 raft_consensus.cc:2229] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:27.607735 15379 raft_consensus.cc:2258] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.608340 15379 ts_tablet_manager.cc:1875] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.609758 15378 ts_tablet_manager.cc:1888] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.609874 15378 log.cc:1196] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/2441f7128305466fbcb0ef64226f34e9
I20240503 07:59:27.610165 15378 ts_tablet_manager.cc:1909] T 2441f7128305466fbcb0ef64226f34e9 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:27.610975 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 2441f7128305466fbcb0ef64226f34e9 (table rebalancer_dynamic_table_new_name [id=5bde7b0605e545aab55481f70236069b]) successfully deleted
I20240503 07:59:27.611315 15377 ts_tablet_manager.cc:1888] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.611433 15379 ts_tablet_manager.cc:1888] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.611452 15377 log.cc:1196] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/2441f7128305466fbcb0ef64226f34e9
I20240503 07:59:27.611521 15379 log.cc:1196] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/2441f7128305466fbcb0ef64226f34e9
I20240503 07:59:27.611768 15377 ts_tablet_manager.cc:1909] T 2441f7128305466fbcb0ef64226f34e9 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:27.611788 15379 ts_tablet_manager.cc:1909] T 2441f7128305466fbcb0ef64226f34e9 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:27.612475 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 2441f7128305466fbcb0ef64226f34e9 (table rebalancer_dynamic_table_new_name [id=5bde7b0605e545aab55481f70236069b]) successfully deleted
I20240503 07:59:27.612489 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 2441f7128305466fbcb0ef64226f34e9 (table rebalancer_dynamic_table_new_name [id=5bde7b0605e545aab55481f70236069b]) successfully deleted
I20240503 07:59:27.657011 6886 raft_consensus.cc:1203] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:27.661461 15060 raft_consensus.cc:482] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.661576 15060 raft_consensus.cc:504] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.662065 15060 leader_election.cc:288] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:27.662390 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ffde5aaa3b4a67a878c77ef73472d6" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:27.662448 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ffde5aaa3b4a67a878c77ef73472d6" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:27.662560 7020 raft_consensus.cc:2454] T 84ffde5aaa3b4a67a878c77ef73472d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:27.662561 7150 raft_consensus.cc:2454] T 84ffde5aaa3b4a67a878c77ef73472d6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:27.662824 9572 leader_election.cc:302] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2; no voters:
I20240503 07:59:27.662981 15060 raft_consensus.cc:2781] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.663056 15060 raft_consensus.cc:482] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.663095 15060 raft_consensus.cc:3037] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.664361 15060 raft_consensus.cc:504] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.664733 15060 leader_election.cc:288] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:27.665030 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ffde5aaa3b4a67a878c77ef73472d6" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:27.665071 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ffde5aaa3b4a67a878c77ef73472d6" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:27.665170 7020 raft_consensus.cc:3037] T 84ffde5aaa3b4a67a878c77ef73472d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.665171 7150 raft_consensus.cc:3037] T 84ffde5aaa3b4a67a878c77ef73472d6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.666205 7150 raft_consensus.cc:2454] T 84ffde5aaa3b4a67a878c77ef73472d6 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:27.666196 7020 raft_consensus.cc:2454] T 84ffde5aaa3b4a67a878c77ef73472d6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:27.666600 9571 leader_election.cc:302] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:27.666792 15060 raft_consensus.cc:2781] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.666882 15060 raft_consensus.cc:686] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:27.667019 15060 consensus_queue.cc:237] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.669356 6745 catalog_manager.cc:5484] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:27.692168 9766 raft_consensus.cc:1203] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:27.706984 9615 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:27.708506 9351 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:27.709216 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:27.709415 6745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:27.711282 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:27.715327 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:27.721909 7260 tablet_service.cc:1467] Processing CreateTablet for tablet b73cfb06b7854cd4a680b31804b7fc1c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b5f5f056cee4aa896e9e0fc31bdf670]), partition=
I20240503 07:59:27.722229 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73cfb06b7854cd4a680b31804b7fc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.723043 9615 tablet_service.cc:1467] Processing CreateTablet for tablet b73cfb06b7854cd4a680b31804b7fc1c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b5f5f056cee4aa896e9e0fc31bdf670]), partition=
I20240503 07:59:27.723287 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73cfb06b7854cd4a680b31804b7fc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.724860 9745 tablet_service.cc:1467] Processing CreateTablet for tablet b73cfb06b7854cd4a680b31804b7fc1c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b5f5f056cee4aa896e9e0fc31bdf670]), partition=
I20240503 07:59:27.725121 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73cfb06b7854cd4a680b31804b7fc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.725677 15236 tablet_bootstrap.cc:492] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:27.726701 15236 tablet_bootstrap.cc:654] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.743142 15397 tablet_bootstrap.cc:492] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:27.745461 15397 tablet_bootstrap.cc:654] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.750641 15236 tablet_bootstrap.cc:492] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:27.750753 15236 ts_tablet_manager.cc:1367] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 07:59:27.751231 15236 raft_consensus.cc:348] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:27.751339 15236 raft_consensus.cc:374] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.751366 15236 raft_consensus.cc:729] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.751477 15236 consensus_queue.cc:260] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:27.751755 15236 ts_tablet_manager.cc:1398] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.760690 15397 tablet_bootstrap.cc:492] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:27.760826 15397 ts_tablet_manager.cc:1367] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20240503 07:59:27.761533 15397 raft_consensus.cc:348] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:27.761682 15397 raft_consensus.cc:374] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.761716 15397 raft_consensus.cc:729] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.761829 15397 consensus_queue.cc:260] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:27.762101 15397 ts_tablet_manager.cc:1398] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 07:59:27.765275 15399 tablet_bootstrap.cc:492] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:27.765911 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:27.768254 9485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:27.777563 15083 consensus_queue.cc:1035] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.782522 6867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:27.784870 15399 tablet_bootstrap.cc:654] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.800247 15404 consensus_queue.cc:1035] T 3f6388b40cba4095b15b1fdbb1f2e9f9 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.800336 15399 tablet_bootstrap.cc:492] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:27.800585 15399 ts_tablet_manager.cc:1367] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20240503 07:59:27.801160 15399 raft_consensus.cc:348] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:27.801281 15399 raft_consensus.cc:374] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.801309 15399 raft_consensus.cc:729] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.801426 15399 consensus_queue.cc:260] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:27.801776 15399 ts_tablet_manager.cc:1398] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.821627 9374 raft_consensus.cc:1203] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:27.837221 7278 consensus_queue.cc:237] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:27.838742 7150 raft_consensus.cc:1261] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:27.839179 9634 raft_consensus.cc:1261] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:27.839161 14839 consensus_queue.cc:1035] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:27.839571 14839 consensus_queue.cc:1035] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:27.842221 14440 raft_consensus.cc:2932] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:27.842917 9634 raft_consensus.cc:2932] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:27.845149 7150 raft_consensus.cc:2932] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:27.845453 6745 catalog_manager.cc:5484] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:27.849709 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fcc4c2eeaec4b359108ec64a6e20da8 with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:27.850140 15431 tablet_replica.cc:330] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:27.850304 15431 raft_consensus.cc:2229] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:27.850560 15431 raft_consensus.cc:2258] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.851173 15431 ts_tablet_manager.cc:1875] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:27.854034 15431 ts_tablet_manager.cc:1888] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:27.854131 15431 log.cc:1196] T 0fcc4c2eeaec4b359108ec64a6e20da8 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0fcc4c2eeaec4b359108ec64a6e20da8
I20240503 07:59:27.854738 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 0fcc4c2eeaec4b359108ec64a6e20da8 (table rebalance_test_table_9 [id=3e242002464141aaa2f77f2115a129ff]) successfully deleted
I20240503 07:59:27.883877 13216 consensus_queue.cc:1035] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.888108 15151 consensus_queue.cc:1035] T c2b5e8db1eec44a7881dde976e8a0a12 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.936440 6886 raft_consensus.cc:1203] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:28.037528 15060 raft_consensus.cc:482] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.037701 15060 raft_consensus.cc:504] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:28.038268 15060 leader_election.cc:288] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:28.038962 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73cfb06b7854cd4a680b31804b7fc1c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:28.039114 9767 raft_consensus.cc:2454] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:28.039163 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73cfb06b7854cd4a680b31804b7fc1c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:28.039310 7278 raft_consensus.cc:2454] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:28.039513 9572 leader_election.cc:302] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:28.039777 15060 raft_consensus.cc:2781] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.039852 15060 raft_consensus.cc:482] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.039887 15060 raft_consensus.cc:3037] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.041015 15060 raft_consensus.cc:504] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:28.041352 15060 leader_election.cc:288] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:28.041704 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73cfb06b7854cd4a680b31804b7fc1c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:28.041841 9767 raft_consensus.cc:3037] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.041872 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73cfb06b7854cd4a680b31804b7fc1c" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:28.041971 7278 raft_consensus.cc:3037] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.042971 7278 raft_consensus.cc:2454] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:28.042971 9767 raft_consensus.cc:2454] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:28.043329 9572 leader_election.cc:302] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:28.043663 15060 raft_consensus.cc:2781] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.043752 15060 raft_consensus.cc:686] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:28.043890 15060 consensus_queue.cc:237] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:28.045997 6746 catalog_manager.cc:5484] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:28.069898 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:28.072826 9351 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:28.074819 7260 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:28.078980 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:28.080332 9615 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:28.082036 6867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:28.093454 15060 consensus_queue.cc:1035] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.108177 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:28.115222 9485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:28.135298 15060 consensus_queue.cc:1035] T 84ffde5aaa3b4a67a878c77ef73472d6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.170382 6886 consensus_queue.cc:237] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:28.172008 9504 raft_consensus.cc:1261] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:28.172192 7020 raft_consensus.cc:1261] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:28.172521 14629 consensus_queue.cc:1035] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:28.172736 14475 consensus_queue.cc:1035] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:28.174803 15506 raft_consensus.cc:2932] T f365955974634bc58e8a0da0fc2563d7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:28.175073 9504 raft_consensus.cc:2932] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:28.175073 7020 raft_consensus.cc:2932] T f365955974634bc58e8a0da0fc2563d7 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:28.178723 6746 catalog_manager.cc:5484] T f365955974634bc58e8a0da0fc2563d7 P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:28.184396 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet f365955974634bc58e8a0da0fc2563d7 with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:28.184836 15507 tablet_replica.cc:330] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:28.185052 15507 raft_consensus.cc:2229] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:28.185281 15507 raft_consensus.cc:2258] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.185945 15507 ts_tablet_manager.cc:1875] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:28.188757 15507 ts_tablet_manager.cc:1888] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:28.188841 15507 log.cc:1196] T f365955974634bc58e8a0da0fc2563d7 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f365955974634bc58e8a0da0fc2563d7
I20240503 07:59:28.189841 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet f365955974634bc58e8a0da0fc2563d7 (table rebalance_test_table_11 [id=9caa58273688477ab18820b0590675ab]) successfully deleted
I20240503 07:59:28.223404 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:28.228350 9767 raft_consensus.cc:1261] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.228350 7278 raft_consensus.cc:1261] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.229089 15060 consensus_queue.cc:1035] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.229393 15060 consensus_queue.cc:1035] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.237452 6746 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet b73cfb06b7854cd4a680b31804b7fc1c (table rebalancer_dynamic_table [id=7b5f5f056cee4aa896e9e0fc31bdf670]). Expected version 1 got 0
I20240503 07:59:28.238740 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:28.239825 14037 mvcc.cc:205] Tried to move back new op lower bound from 7023506097058586624 to 7023506096309198848. Current Snapshot: MvccSnapshot[applied={T|T < 7023506097058586624}]
I20240503 07:59:28.240857 6745 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet b73cfb06b7854cd4a680b31804b7fc1c (table rebalancer_dynamic_table [id=7b5f5f056cee4aa896e9e0fc31bdf670]). Expected version 1 got 0
I20240503 07:59:28.243196 6745 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet b73cfb06b7854cd4a680b31804b7fc1c (table rebalancer_dynamic_table [id=7b5f5f056cee4aa896e9e0fc31bdf670]). Expected version 1 got 0
I20240503 07:59:28.244268 6742 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet b73cfb06b7854cd4a680b31804b7fc1c (table rebalancer_dynamic_table [id=7b5f5f056cee4aa896e9e0fc31bdf670]). Expected version 1 got 0
I20240503 07:59:28.244390 15532 tablet.cc:1722] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.244750 15532 tablet.cc:1620] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:28.249951 15555 tablet.cc:1722] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.250419 15555 tablet.cc:1620] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:28.251461 15556 tablet.cc:1722] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.251672 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 085ffbb3db1947e6b244e41a0a23e4de (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=1aa23fc281844edb86eaed8d479e1d4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.251801 15556 tablet.cc:1620] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:28.251780 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 20680ae4502042068d823f875c729166 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=1aa23fc281844edb86eaed8d479e1d4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.251958 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 20680ae4502042068d823f875c729166 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=1aa23fc281844edb86eaed8d479e1d4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.252019 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 085ffbb3db1947e6b244e41a0a23e4de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.252019 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20680ae4502042068d823f875c729166. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.252238 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20680ae4502042068d823f875c729166. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.254666 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 20680ae4502042068d823f875c729166 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=1aa23fc281844edb86eaed8d479e1d4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.254940 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20680ae4502042068d823f875c729166. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.255565 7131 tablet_service.cc:1467] Processing CreateTablet for tablet dee8b0dc9f0348a3aadd366671a2b0cc (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=1aa23fc281844edb86eaed8d479e1d4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.255856 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee8b0dc9f0348a3aadd366671a2b0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.256582 9483 tablet_service.cc:1467] Processing CreateTablet for tablet dee8b0dc9f0348a3aadd366671a2b0cc (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=1aa23fc281844edb86eaed8d479e1d4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.256867 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee8b0dc9f0348a3aadd366671a2b0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.257355 15397 tablet_bootstrap.cc:492] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:28.258620 9747 tablet_service.cc:1467] Processing CreateTablet for tablet dee8b0dc9f0348a3aadd366671a2b0cc (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=1aa23fc281844edb86eaed8d479e1d4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.258874 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee8b0dc9f0348a3aadd366671a2b0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.258991 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 085ffbb3db1947e6b244e41a0a23e4de (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=1aa23fc281844edb86eaed8d479e1d4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.259217 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 085ffbb3db1947e6b244e41a0a23e4de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.260926 15399 tablet_bootstrap.cc:492] T 20680ae4502042068d823f875c729166 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:28.261826 15399 tablet_bootstrap.cc:654] T 20680ae4502042068d823f875c729166 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.267570 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=7b5f5f056cee4aa896e9e0fc31bdf670] alter complete (version 1)
I20240503 07:59:28.267632 15397 tablet_bootstrap.cc:654] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.270223 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 085ffbb3db1947e6b244e41a0a23e4de (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=1aa23fc281844edb86eaed8d479e1d4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.270576 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 085ffbb3db1947e6b244e41a0a23e4de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.271447 15557 tablet_bootstrap.cc:492] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:28.272066 15558 tablet_bootstrap.cc:492] T 20680ae4502042068d823f875c729166 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:28.272639 15557 tablet_bootstrap.cc:654] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.273351 15558 tablet_bootstrap.cc:654] T 20680ae4502042068d823f875c729166 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.273887 15397 tablet_bootstrap.cc:492] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:28.273995 15397 ts_tablet_manager.cc:1367] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240503 07:59:28.274567 15397 raft_consensus.cc:348] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:28.274696 15397 raft_consensus.cc:374] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.274765 15397 raft_consensus.cc:729] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.275065 15558 tablet_bootstrap.cc:492] T 20680ae4502042068d823f875c729166 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:28.275151 15558 ts_tablet_manager.cc:1367] T 20680ae4502042068d823f875c729166 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:28.275705 15558 raft_consensus.cc:348] T 20680ae4502042068d823f875c729166 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:28.275882 15397 consensus_queue.cc:260] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:28.276286 15397 ts_tablet_manager.cc:1398] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:28.277040 15557 tablet_bootstrap.cc:492] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:28.277184 15557 ts_tablet_manager.cc:1367] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:28.277803 15397 tablet_bootstrap.cc:492] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:28.278158 15399 tablet_bootstrap.cc:492] T 20680ae4502042068d823f875c729166 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:28.278262 15399 ts_tablet_manager.cc:1367] T 20680ae4502042068d823f875c729166 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.000s
I20240503 07:59:28.278553 15558 raft_consensus.cc:374] T 20680ae4502042068d823f875c729166 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.278609 15558 raft_consensus.cc:729] T 20680ae4502042068d823f875c729166 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.278759 15558 consensus_queue.cc:260] T 20680ae4502042068d823f875c729166 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:28.278757 15399 raft_consensus.cc:348] T 20680ae4502042068d823f875c729166 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:28.278872 15399 raft_consensus.cc:374] T 20680ae4502042068d823f875c729166 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.278905 15399 raft_consensus.cc:729] T 20680ae4502042068d823f875c729166 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.279028 15399 consensus_queue.cc:260] T 20680ae4502042068d823f875c729166 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:28.279346 15399 ts_tablet_manager.cc:1398] T 20680ae4502042068d823f875c729166 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:28.279526 15399 tablet_bootstrap.cc:492] T dee8b0dc9f0348a3aadd366671a2b0cc P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:28.280535 15399 tablet_bootstrap.cc:654] T dee8b0dc9f0348a3aadd366671a2b0cc P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.281512 15397 tablet_bootstrap.cc:654] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.283703 15557 raft_consensus.cc:348] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:28.283856 15557 raft_consensus.cc:374] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.283893 15557 raft_consensus.cc:729] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.284029 15557 consensus_queue.cc:260] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:28.284731 15557 ts_tablet_manager.cc:1398] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:28.285663 15559 tablet_bootstrap.cc:492] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:28.287175 15559 tablet_bootstrap.cc:654] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.287695 15399 tablet_bootstrap.cc:492] T dee8b0dc9f0348a3aadd366671a2b0cc P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:28.287812 15399 ts_tablet_manager.cc:1367] T dee8b0dc9f0348a3aadd366671a2b0cc P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 07:59:28.288319 15399 raft_consensus.cc:348] T dee8b0dc9f0348a3aadd366671a2b0cc P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:28.288444 15399 raft_consensus.cc:374] T dee8b0dc9f0348a3aadd366671a2b0cc P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.288480 15399 raft_consensus.cc:729] T dee8b0dc9f0348a3aadd366671a2b0cc P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.288599 15399 consensus_queue.cc:260] T dee8b0dc9f0348a3aadd366671a2b0cc P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:28.288897 15399 ts_tablet_manager.cc:1398] T dee8b0dc9f0348a3aadd366671a2b0cc P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:28.291595 15397 tablet_bootstrap.cc:492] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:28.291749 15397 ts_tablet_manager.cc:1367] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:28.292305 15397 raft_consensus.cc:348] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:28.292493 15397 raft_consensus.cc:374] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.292546 15397 raft_consensus.cc:729] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.292662 15397 consensus_queue.cc:260] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:28.292946 15397 ts_tablet_manager.cc:1398] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.302055 15558 ts_tablet_manager.cc:1398] T 20680ae4502042068d823f875c729166 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20240503 07:59:28.302361 15558 tablet_bootstrap.cc:492] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:28.306496 15558 tablet_bootstrap.cc:654] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.312259 15558 tablet_bootstrap.cc:492] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:28.312373 15558 ts_tablet_manager.cc:1367] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:28.312852 15558 raft_consensus.cc:348] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:28.312971 15558 raft_consensus.cc:374] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.312992 15558 raft_consensus.cc:729] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.313129 15558 consensus_queue.cc:260] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:28.314184 15558 ts_tablet_manager.cc:1398] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:28.314441 15558 tablet_bootstrap.cc:492] T 085ffbb3db1947e6b244e41a0a23e4de P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:28.324434 15559 tablet_bootstrap.cc:492] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:28.325637 15558 tablet_bootstrap.cc:654] T 085ffbb3db1947e6b244e41a0a23e4de P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.325881 15559 ts_tablet_manager.cc:1367] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240503 07:59:28.326534 15559 raft_consensus.cc:348] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:28.326690 15559 raft_consensus.cc:374] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.326740 15559 raft_consensus.cc:729] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.326898 15559 consensus_queue.cc:260] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:28.327220 15559 ts_tablet_manager.cc:1398] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.328701 15558 tablet_bootstrap.cc:492] T 085ffbb3db1947e6b244e41a0a23e4de P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:28.328814 15558 ts_tablet_manager.cc:1367] T 085ffbb3db1947e6b244e41a0a23e4de P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 07:59:28.329282 15558 raft_consensus.cc:348] T 085ffbb3db1947e6b244e41a0a23e4de P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:28.329420 15558 raft_consensus.cc:374] T 085ffbb3db1947e6b244e41a0a23e4de P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.329452 15558 raft_consensus.cc:729] T 085ffbb3db1947e6b244e41a0a23e4de P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.329569 15558 consensus_queue.cc:260] T 085ffbb3db1947e6b244e41a0a23e4de P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:28.329871 15558 ts_tablet_manager.cc:1398] T 085ffbb3db1947e6b244e41a0a23e4de P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.339360 9615 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:28.343562 15471 raft_consensus.cc:482] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.343983 15471 raft_consensus.cc:504] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:28.344543 15471 leader_election.cc:288] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:28.346766 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee8b0dc9f0348a3aadd366671a2b0cc" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:28.346933 7149 raft_consensus.cc:2454] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:28.347256 9441 leader_election.cc:302] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:28.347646 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:28.348831 15471 raft_consensus.cc:2781] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.348898 15471 raft_consensus.cc:482] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.348934 15471 raft_consensus.cc:3037] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.349426 9351 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:28.350139 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee8b0dc9f0348a3aadd366671a2b0cc" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:28.350201 15471 raft_consensus.cc:504] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:28.350312 9766 raft_consensus.cc:2454] T dee8b0dc9f0348a3aadd366671a2b0cc P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:28.352681 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:28.354274 15471 leader_election.cc:288] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:28.355832 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee8b0dc9f0348a3aadd366671a2b0cc" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:28.355985 7149 raft_consensus.cc:3037] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.357060 7149 raft_consensus.cc:2454] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:28.358633 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee8b0dc9f0348a3aadd366671a2b0cc" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:28.358764 9766 raft_consensus.cc:3037] T dee8b0dc9f0348a3aadd366671a2b0cc P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.358834 15425 raft_consensus.cc:482] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.358937 15425 raft_consensus.cc:504] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:28.359424 15425 leader_election.cc:288] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:28.359750 9766 raft_consensus.cc:2454] T dee8b0dc9f0348a3aadd366671a2b0cc P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:28.360405 9441 leader_election.cc:302] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:28.360842 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20680ae4502042068d823f875c729166" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:28.360970 9504 raft_consensus.cc:2454] T 20680ae4502042068d823f875c729166 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:28.361981 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20680ae4502042068d823f875c729166" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:28.362032 7218 leader_election.cc:302] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:28.362109 9766 raft_consensus.cc:2454] T 20680ae4502042068d823f875c729166 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:28.362206 15425 raft_consensus.cc:2781] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.362254 15425 raft_consensus.cc:482] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.362288 15425 raft_consensus.cc:3037] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.362705 15471 raft_consensus.cc:2781] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.362797 15471 raft_consensus.cc:686] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:28.362955 6867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:28.362928 15471 consensus_queue.cc:237] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:28.363394 15425 raft_consensus.cc:504] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:28.363731 15425 leader_election.cc:288] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:28.366475 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20680ae4502042068d823f875c729166" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:28.366596 9504 raft_consensus.cc:3037] T 20680ae4502042068d823f875c729166 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.367642 9504 raft_consensus.cc:2454] T 20680ae4502042068d823f875c729166 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:28.370019 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20680ae4502042068d823f875c729166" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:28.370152 9766 raft_consensus.cc:3037] T 20680ae4502042068d823f875c729166 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.371162 9766 raft_consensus.cc:2454] T 20680ae4502042068d823f875c729166 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:28.374178 7218 leader_election.cc:302] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:28.376559 15425 raft_consensus.cc:2781] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.376662 15425 raft_consensus.cc:686] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:28.376798 15425 consensus_queue.cc:237] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:28.378311 6745 catalog_manager.cc:5484] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:28.379037 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240503 07:59:28.383116 6936 tablet.cc:2369] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:28.384017 6746 catalog_manager.cc:5484] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:28.385154 7260 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:28.391611 15532 tablet.cc:1722] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:28.391980 15532 tablet.cc:1620] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:28.392565 15555 tablet.cc:1722] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:28.392860 15555 tablet.cc:1620] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:28.392946 15556 tablet.cc:1722] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:28.393241 15556 tablet.cc:1620] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:28.399132 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=7b5f5f056cee4aa896e9e0fc31bdf670] alter complete (version 2)
I20240503 07:59:28.402277 9747 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:28.402281 9485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:28.467772 9374 consensus_queue.cc:237] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:28.469163 7149 raft_consensus.cc:1261] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:28.469175 6886 raft_consensus.cc:1261] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:28.469609 15546 consensus_queue.cc:1035] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:28.469818 15549 consensus_queue.cc:1035] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:28.471758 15546 raft_consensus.cc:2932] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:28.472755 6886 raft_consensus.cc:2932] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:28.474510 7149 raft_consensus.cc:2932] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:28.475131 6745 catalog_manager.cc:5484] T 21f55eb986bd4b7bbda10e683f3bfcad P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:28.479377 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet 21f55eb986bd4b7bbda10e683f3bfcad with delete_type TABLET_DATA_TOMBSTONED (TS cb73994515af4880b05383e78368bb4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:28.479844 15605 tablet_replica.cc:330] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:28.480099 15605 raft_consensus.cc:2229] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:28.480351 15605 raft_consensus.cc:2258] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.480842 15605 ts_tablet_manager.cc:1875] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:28.483474 15605 ts_tablet_manager.cc:1888] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:28.483563 15605 log.cc:1196] T 21f55eb986bd4b7bbda10e683f3bfcad P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/21f55eb986bd4b7bbda10e683f3bfcad
I20240503 07:59:28.484088 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 21f55eb986bd4b7bbda10e683f3bfcad (table rebalancer_extra_table_22 [id=52f5ffbd2adb439692c9dea927b73abb]) successfully deleted
W20240503 07:59:28.496552 7329 tablet.cc:2369] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:28.507843 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:28.515307 15532 tablet.cc:1722] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:28.515412 15556 tablet.cc:1722] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:28.519129 15555 tablet.cc:1722] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:28.521133 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=7b5f5f056cee4aa896e9e0fc31bdf670] alter complete (version 3)
W20240503 07:59:28.538161 7198 tablet.cc:2369] T dee8b0dc9f0348a3aadd366671a2b0cc P 00a3f8e159c64327832d79fb32cdcca8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:28.558071 15425 raft_consensus.cc:482] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.558218 15425 raft_consensus.cc:504] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:28.558697 15425 leader_election.cc:288] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:28.559062 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085ffbb3db1947e6b244e41a0a23e4de" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:28.559121 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085ffbb3db1947e6b244e41a0a23e4de" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:28.559235 6888 raft_consensus.cc:2454] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:28.559264 9504 raft_consensus.cc:2454] T 085ffbb3db1947e6b244e41a0a23e4de P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:28.559572 7215 leader_election.cc:302] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:28.559780 15425 raft_consensus.cc:2781] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.559847 15425 raft_consensus.cc:482] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.559885 15425 raft_consensus.cc:3037] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.561093 15425 raft_consensus.cc:504] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:28.561479 15425 leader_election.cc:288] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:28.562016 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085ffbb3db1947e6b244e41a0a23e4de" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:28.562146 9504 raft_consensus.cc:3037] T 085ffbb3db1947e6b244e41a0a23e4de P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.563318 9504 raft_consensus.cc:2454] T 085ffbb3db1947e6b244e41a0a23e4de P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:28.563746 7218 leader_election.cc:302] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:28.564137 14440 raft_consensus.cc:2781] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.564216 14440 raft_consensus.cc:686] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:28.564357 14440 consensus_queue.cc:237] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:28.566247 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085ffbb3db1947e6b244e41a0a23e4de" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:28.566419 6888 raft_consensus.cc:3037] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.567108 6743 catalog_manager.cc:5484] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:28.567550 6888 raft_consensus.cc:2454] T 085ffbb3db1947e6b244e41a0a23e4de P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:28.596519 9615 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:28.599747 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:28.609622 6999 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:28.612663 9351 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:28.622128 6867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:28.625175 6743 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:28.625347 6743 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:28.627944 6743 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet b73cfb06b7854cd4a680b31804b7fc1c
I20240503 07:59:28.629968 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet b73cfb06b7854cd4a680b31804b7fc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:28 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:28.630498 9747 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:28.630693 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet b73cfb06b7854cd4a680b31804b7fc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:28 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:28.630869 15507 tablet_replica.cc:330] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:28.630892 9745 tablet_service.cc:1514] Processing DeleteTablet for tablet b73cfb06b7854cd4a680b31804b7fc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:28 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:28.631023 15507 raft_consensus.cc:2229] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:28.631166 15507 raft_consensus.cc:2258] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.631994 15507 ts_tablet_manager.cc:1875] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:28.636109 15507 ts_tablet_manager.cc:1888] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:28.636263 15507 log.cc:1196] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b73cfb06b7854cd4a680b31804b7fc1c
I20240503 07:59:28.636602 15507 ts_tablet_manager.cc:1909] T b73cfb06b7854cd4a680b31804b7fc1c P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:28.637367 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet b73cfb06b7854cd4a680b31804b7fc1c (table rebalancer_dynamic_table_new_name [id=7b5f5f056cee4aa896e9e0fc31bdf670]) successfully deleted
I20240503 07:59:28.639636 15655 tablet_replica.cc:330] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:28.639797 15655 raft_consensus.cc:2229] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:28.640069 15655 raft_consensus.cc:2258] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.640645 15655 ts_tablet_manager.cc:1875] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:28.642818 15655 ts_tablet_manager.cc:1888] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:28.642891 15655 log.cc:1196] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/b73cfb06b7854cd4a680b31804b7fc1c
I20240503 07:59:28.643477 15655 ts_tablet_manager.cc:1909] T b73cfb06b7854cd4a680b31804b7fc1c P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:28.644212 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet b73cfb06b7854cd4a680b31804b7fc1c (table rebalancer_dynamic_table_new_name [id=7b5f5f056cee4aa896e9e0fc31bdf670]) successfully deleted
I20240503 07:59:28.651759 15654 tablet_replica.cc:330] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:28.652045 15654 raft_consensus.cc:2229] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:28.652307 15654 raft_consensus.cc:2258] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.652962 15654 ts_tablet_manager.cc:1875] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:28.654002 7260 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:28.655022 9485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:28.659909 15654 ts_tablet_manager.cc:1888] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:28.660223 15654 log.cc:1196] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/b73cfb06b7854cd4a680b31804b7fc1c
I20240503 07:59:28.660548 15654 ts_tablet_manager.cc:1909] T b73cfb06b7854cd4a680b31804b7fc1c P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:28.661376 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet b73cfb06b7854cd4a680b31804b7fc1c (table rebalancer_dynamic_table_new_name [id=7b5f5f056cee4aa896e9e0fc31bdf670]) successfully deleted
I20240503 07:59:28.718888 6886 consensus_queue.cc:237] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:28.720219 9635 raft_consensus.cc:1261] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:28.720350 7020 raft_consensus.cc:1261] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:28.720754 15506 consensus_queue.cc:1035] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:28.722837 15506 raft_consensus.cc:2932] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:28.723212 7020 raft_consensus.cc:2932] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:28.724367 15506 consensus_queue.cc:1035] T cb2190946fb04a6392097f9588ffda6b P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:28.725627 9635 raft_consensus.cc:2932] T cb2190946fb04a6392097f9588ffda6b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:28.726300 6743 catalog_manager.cc:5484] T cb2190946fb04a6392097f9588ffda6b P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 4 to 5, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:28.730579 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet cb2190946fb04a6392097f9588ffda6b with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:28.730829 15507 tablet_replica.cc:330] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:28.730765 6743 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:28.730984 15507 raft_consensus.cc:2229] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:28.731138 15507 raft_consensus.cc:2258] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.731837 15507 ts_tablet_manager.cc:1875] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:28.736689 15507 ts_tablet_manager.cc:1888] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:28.736904 15507 log.cc:1196] T cb2190946fb04a6392097f9588ffda6b P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/cb2190946fb04a6392097f9588ffda6b
I20240503 07:59:28.737635 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet cb2190946fb04a6392097f9588ffda6b (table rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]) successfully deleted
I20240503 07:59:28.744769 7260 tablet_service.cc:1467] Processing CreateTablet for tablet d338ae796f6d402d87d2ec1e3e34e416 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=695e494ba1d7474fb2dc6d2794575a22]), partition=
I20240503 07:59:28.745069 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d338ae796f6d402d87d2ec1e3e34e416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.745071 9351 tablet_service.cc:1467] Processing CreateTablet for tablet d338ae796f6d402d87d2ec1e3e34e416 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=695e494ba1d7474fb2dc6d2794575a22]), partition=
I20240503 07:59:28.745309 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d338ae796f6d402d87d2ec1e3e34e416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.747920 6999 tablet_service.cc:1467] Processing CreateTablet for tablet d338ae796f6d402d87d2ec1e3e34e416 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=695e494ba1d7474fb2dc6d2794575a22]), partition=
I20240503 07:59:28.747968 15397 tablet_bootstrap.cc:492] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:28.748157 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d338ae796f6d402d87d2ec1e3e34e416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.748817 15397 tablet_bootstrap.cc:654] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.749017 15686 tablet_bootstrap.cc:492] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:28.749893 15686 tablet_bootstrap.cc:654] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.750666 15689 tablet_bootstrap.cc:492] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:28.750885 15686 tablet_bootstrap.cc:492] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:28.750972 15686 ts_tablet_manager.cc:1367] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:28.751397 15686 raft_consensus.cc:348] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:28.751497 15686 raft_consensus.cc:374] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.751523 15686 raft_consensus.cc:729] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.751633 15686 consensus_queue.cc:260] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:28.751901 15686 ts_tablet_manager.cc:1398] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.752173 15689 tablet_bootstrap.cc:654] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.752554 15397 tablet_bootstrap.cc:492] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:28.752641 15397 ts_tablet_manager.cc:1367] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 07:59:28.753105 15397 raft_consensus.cc:348] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:28.753233 15397 raft_consensus.cc:374] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.753268 15397 raft_consensus.cc:729] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.753335 15689 tablet_bootstrap.cc:492] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:28.753413 15689 ts_tablet_manager.cc:1367] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:28.753388 15397 consensus_queue.cc:260] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:28.753656 15397 ts_tablet_manager.cc:1398] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.753855 15689 raft_consensus.cc:348] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:28.753954 15689 raft_consensus.cc:374] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.753988 15689 raft_consensus.cc:729] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.754123 15689 consensus_queue.cc:260] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:28.754405 15689 ts_tablet_manager.cc:1398] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.796916 14839 consensus_queue.cc:1035] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.802492 14839 consensus_queue.cc:1035] T 20680ae4502042068d823f875c729166 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:28.844938 7067 tablet.cc:2369] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:28.927093 15304 consensus_queue.cc:1035] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.930223 15304 consensus_queue.cc:1035] T dee8b0dc9f0348a3aadd366671a2b0cc P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.971047 15726 raft_consensus.cc:482] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.971175 15726 raft_consensus.cc:504] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:28.971585 15726 leader_election.cc:288] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:28.971941 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d338ae796f6d402d87d2ec1e3e34e416" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:28.971979 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d338ae796f6d402d87d2ec1e3e34e416" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:28.972100 9374 raft_consensus.cc:2454] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:28.972126 7282 raft_consensus.cc:2454] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:28.972395 6956 leader_election.cc:302] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:28.972566 15726 raft_consensus.cc:2781] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.972637 15726 raft_consensus.cc:482] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.972674 15726 raft_consensus.cc:3037] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.973735 15726 raft_consensus.cc:504] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:28.974102 15726 leader_election.cc:288] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:28.974519 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d338ae796f6d402d87d2ec1e3e34e416" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:28.974656 7282 raft_consensus.cc:3037] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.975520 7282 raft_consensus.cc:2454] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:28.975862 6956 leader_election.cc:302] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:28.976027 15726 raft_consensus.cc:2781] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.976120 15726 raft_consensus.cc:686] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:28.976266 15726 consensus_queue.cc:237] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:28.978943 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d338ae796f6d402d87d2ec1e3e34e416" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:28.978940 14839 consensus_queue.cc:1035] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.979077 9374 raft_consensus.cc:3037] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.979635 6746 catalog_manager.cc:5484] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:28.980202 9374 raft_consensus.cc:2454] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:28.983498 14440 consensus_queue.cc:1035] T 085ffbb3db1947e6b244e41a0a23e4de P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.003072 9615 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:29.003350 7131 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:29.018441 7260 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:29.026764 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:29.034540 6999 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:29.047118 6867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:29.055774 9485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:29.062551 9351 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:29.087975 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:29.089004 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:29.093060 7282 raft_consensus.cc:1261] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:29.093168 9374 raft_consensus.cc:1261] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:29.093514 15487 consensus_queue.cc:1035] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.094820 15487 consensus_queue.cc:1035] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.096314 6746 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet d338ae796f6d402d87d2ec1e3e34e416 (table rebalancer_dynamic_table [id=695e494ba1d7474fb2dc6d2794575a22]). Expected version 1 got 0
I20240503 07:59:29.096920 15748 tablet.cc:1722] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:29.097440 15748 tablet.cc:1620] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: MemRowSet was empty: no flush needed.
I20240503 07:59:29.097992 15749 tablet.cc:1722] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:29.098407 15749 tablet.cc:1620] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:29.099617 6742 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet d338ae796f6d402d87d2ec1e3e34e416 (table rebalancer_dynamic_table [id=695e494ba1d7474fb2dc6d2794575a22]). Expected version 1 got 0
I20240503 07:59:29.103519 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7a9734f93e4991a0764c9bb116dac8 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=71d5cd8788a344cd8eeb0e6b235ebddf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.103555 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7a9734f93e4991a0764c9bb116dac8 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=71d5cd8788a344cd8eeb0e6b235ebddf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.103832 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7a9734f93e4991a0764c9bb116dac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.103847 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7a9734f93e4991a0764c9bb116dac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.103883 6739 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet d338ae796f6d402d87d2ec1e3e34e416 (table rebalancer_dynamic_table [id=695e494ba1d7474fb2dc6d2794575a22]). Expected version 1 got 0
I20240503 07:59:29.104274 9351 tablet_service.cc:1467] Processing CreateTablet for tablet bc3d64c714fd471b9f055af7cb172a39 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=71d5cd8788a344cd8eeb0e6b235ebddf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.104521 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3d64c714fd471b9f055af7cb172a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.104561 9615 tablet_service.cc:1467] Processing CreateTablet for tablet cab217306da64dd7bb123e7fb890d103 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=71d5cd8788a344cd8eeb0e6b235ebddf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.104826 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab217306da64dd7bb123e7fb890d103. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.105079 9483 tablet_service.cc:1467] Processing CreateTablet for tablet cab217306da64dd7bb123e7fb890d103 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=71d5cd8788a344cd8eeb0e6b235ebddf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.105304 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab217306da64dd7bb123e7fb890d103. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.104274 9350 tablet_service.cc:1467] Processing CreateTablet for tablet cab217306da64dd7bb123e7fb890d103 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=71d5cd8788a344cd8eeb0e6b235ebddf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.107879 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab217306da64dd7bb123e7fb890d103. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.107928 7260 tablet_service.cc:1467] Processing CreateTablet for tablet bc3d64c714fd471b9f055af7cb172a39 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=71d5cd8788a344cd8eeb0e6b235ebddf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.108173 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3d64c714fd471b9f055af7cb172a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.110363 15397 tablet_bootstrap.cc:492] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:29.111078 6742 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=695e494ba1d7474fb2dc6d2794575a22] alter complete (version 1)
I20240503 07:59:29.111346 15397 tablet_bootstrap.cc:654] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.111809 6999 tablet_service.cc:1467] Processing CreateTablet for tablet bc3d64c714fd471b9f055af7cb172a39 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=71d5cd8788a344cd8eeb0e6b235ebddf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.111892 15753 tablet_bootstrap.cc:492] T 9c7a9734f93e4991a0764c9bb116dac8 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:29.112108 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3d64c714fd471b9f055af7cb172a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.112466 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7a9734f93e4991a0764c9bb116dac8 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=71d5cd8788a344cd8eeb0e6b235ebddf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.112707 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7a9734f93e4991a0764c9bb116dac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.112923 15753 tablet_bootstrap.cc:654] T 9c7a9734f93e4991a0764c9bb116dac8 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.113633 15754 tablet_bootstrap.cc:492] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:29.114686 15754 tablet_bootstrap.cc:654] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.115427 15752 tablet_bootstrap.cc:492] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:29.117841 15689 tablet_bootstrap.cc:492] T 9c7a9734f93e4991a0764c9bb116dac8 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:29.118472 15752 tablet_bootstrap.cc:654] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.118703 15689 tablet_bootstrap.cc:654] T 9c7a9734f93e4991a0764c9bb116dac8 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.118686 6746 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet d338ae796f6d402d87d2ec1e3e34e416 (table rebalancer_dynamic_table [id=695e494ba1d7474fb2dc6d2794575a22]). Expected version 1 got 0
I20240503 07:59:29.120139 15752 tablet_bootstrap.cc:492] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:29.120240 15752 ts_tablet_manager.cc:1367] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 07:59:29.120771 15752 raft_consensus.cc:348] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.121047 15752 raft_consensus.cc:374] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.121142 15689 tablet_bootstrap.cc:492] T 9c7a9734f93e4991a0764c9bb116dac8 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:29.121141 15752 raft_consensus.cc:729] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.121129 15758 tablet.cc:1722] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:29.121237 15689 ts_tablet_manager.cc:1367] T 9c7a9734f93e4991a0764c9bb116dac8 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:29.121294 15752 consensus_queue.cc:260] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.121614 15752 ts_tablet_manager.cc:1398] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:29.121694 15689 raft_consensus.cc:348] T 9c7a9734f93e4991a0764c9bb116dac8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.121798 15689 raft_consensus.cc:374] T 9c7a9734f93e4991a0764c9bb116dac8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.121824 15689 raft_consensus.cc:729] T 9c7a9734f93e4991a0764c9bb116dac8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.121944 15753 tablet_bootstrap.cc:492] T 9c7a9734f93e4991a0764c9bb116dac8 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:29.121929 15689 consensus_queue.cc:260] T 9c7a9734f93e4991a0764c9bb116dac8 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.122017 15753 ts_tablet_manager.cc:1367] T 9c7a9734f93e4991a0764c9bb116dac8 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:29.122205 15689 ts_tablet_manager.cc:1398] T 9c7a9734f93e4991a0764c9bb116dac8 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.122398 15689 tablet_bootstrap.cc:492] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:29.122512 15753 raft_consensus.cc:348] T 9c7a9734f93e4991a0764c9bb116dac8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.122615 15753 raft_consensus.cc:374] T 9c7a9734f93e4991a0764c9bb116dac8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.122644 15753 raft_consensus.cc:729] T 9c7a9734f93e4991a0764c9bb116dac8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.122756 15753 consensus_queue.cc:260] T 9c7a9734f93e4991a0764c9bb116dac8 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.123016 15753 ts_tablet_manager.cc:1398] T 9c7a9734f93e4991a0764c9bb116dac8 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.123193 15753 tablet_bootstrap.cc:492] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:29.123286 15689 tablet_bootstrap.cc:654] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.124362 15753 tablet_bootstrap.cc:654] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.124774 15686 tablet_bootstrap.cc:492] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:29.125445 15758 tablet.cc:1620] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:29.125659 15686 tablet_bootstrap.cc:654] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.127396 15397 tablet_bootstrap.cc:492] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:29.127498 15397 ts_tablet_manager.cc:1367] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 07:59:29.127677 15754 tablet_bootstrap.cc:492] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:29.127677 15686 tablet_bootstrap.cc:492] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:29.127759 15686 ts_tablet_manager.cc:1367] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:29.127764 15754 ts_tablet_manager.cc:1367] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:29.128000 15397 raft_consensus.cc:348] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:29.128123 15397 raft_consensus.cc:374] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.128155 15397 raft_consensus.cc:729] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.128192 15686 raft_consensus.cc:348] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:29.128291 15686 raft_consensus.cc:374] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.128229 15754 raft_consensus.cc:348] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:29.128320 15686 raft_consensus.cc:729] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.128288 15397 consensus_queue.cc:260] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:29.128346 15754 raft_consensus.cc:374] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.128376 15754 raft_consensus.cc:729] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.128433 15686 consensus_queue.cc:260] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:29.128479 15754 consensus_queue.cc:260] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:29.128715 15686 ts_tablet_manager.cc:1398] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:29.128731 15754 ts_tablet_manager.cc:1398] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.129637 15397 ts_tablet_manager.cc:1398] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:29.130041 15686 tablet_bootstrap.cc:492] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:29.130074 15727 raft_consensus.cc:482] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.130169 15727 raft_consensus.cc:504] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.130687 15727 leader_election.cc:288] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:29.131234 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7a9734f93e4991a0764c9bb116dac8" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:29.131357 9504 raft_consensus.cc:2454] T 9c7a9734f93e4991a0764c9bb116dac8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:29.132746 7087 leader_election.cc:302] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:29.132948 15727 raft_consensus.cc:2781] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.133013 15727 raft_consensus.cc:482] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.133046 15727 raft_consensus.cc:3037] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.134460 15727 raft_consensus.cc:504] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.134647 15686 tablet_bootstrap.cc:654] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.134871 15727 leader_election.cc:288] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:29.135329 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7a9734f93e4991a0764c9bb116dac8" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:29.135594 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7a9734f93e4991a0764c9bb116dac8" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:29.135711 7019 raft_consensus.cc:3037] T 9c7a9734f93e4991a0764c9bb116dac8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.135700 9504 raft_consensus.cc:3037] T 9c7a9734f93e4991a0764c9bb116dac8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.136528 9504 raft_consensus.cc:2454] T 9c7a9734f93e4991a0764c9bb116dac8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:29.136538 7019 raft_consensus.cc:2454] T 9c7a9734f93e4991a0764c9bb116dac8 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:29.136936 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7a9734f93e4991a0764c9bb116dac8" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:29.137048 7020 raft_consensus.cc:2362] T 9c7a9734f93e4991a0764c9bb116dac8 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1. Re-sending same reply.
I20240503 07:59:29.137228 7086 leader_election.cc:302] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:29.137535 15727 raft_consensus.cc:2781] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.137630 15727 raft_consensus.cc:686] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:29.137782 15727 consensus_queue.cc:237] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.142489 15753 tablet_bootstrap.cc:492] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:29.142475 6746 catalog_manager.cc:5484] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:29.142603 15753 ts_tablet_manager.cc:1367] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 07:59:29.142939 15686 tablet_bootstrap.cc:492] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:29.143033 15686 ts_tablet_manager.cc:1367] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:29.143142 15753 raft_consensus.cc:348] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:29.143256 15753 raft_consensus.cc:374] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.143285 15753 raft_consensus.cc:729] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.143445 15689 tablet_bootstrap.cc:492] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:29.143411 15753 consensus_queue.cc:260] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:29.143410 15686 raft_consensus.cc:348] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:29.143507 15686 raft_consensus.cc:374] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.143525 15689 ts_tablet_manager.cc:1367] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 07:59:29.143530 15686 raft_consensus.cc:729] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.143652 15686 consensus_queue.cc:260] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:29.143712 15753 ts_tablet_manager.cc:1398] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:29.143913 15686 ts_tablet_manager.cc:1398] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.144093 15689 raft_consensus.cc:348] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:29.144203 15689 raft_consensus.cc:374] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.144245 15689 raft_consensus.cc:729] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.144361 15689 consensus_queue.cc:260] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:29.144613 15689 ts_tablet_manager.cc:1398] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.162379 7282 consensus_queue.cc:237] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:29.163702 7149 raft_consensus.cc:1261] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:29.164093 9766 raft_consensus.cc:1261] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:29.164096 14440 consensus_queue.cc:1035] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:29.164453 14440 consensus_queue.cc:1035] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:29.165928 14839 raft_consensus.cc:2932] T 50169f7d514d4fe3a4f846faaed03325 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:29.166329 7149 raft_consensus.cc:2932] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:29.167209 9766 raft_consensus.cc:2932] T 50169f7d514d4fe3a4f846faaed03325 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:29.170085 6739 catalog_manager.cc:5484] T 50169f7d514d4fe3a4f846faaed03325 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index 2 to 3, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:29.173344 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 50169f7d514d4fe3a4f846faaed03325 with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:29.173810 15786 tablet_replica.cc:330] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:29.173969 15786 raft_consensus.cc:2229] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:29.174130 15786 raft_consensus.cc:2258] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.174588 15786 ts_tablet_manager.cc:1875] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:29.177278 15786 ts_tablet_manager.cc:1888] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:29.177490 15786 log.cc:1196] T 50169f7d514d4fe3a4f846faaed03325 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/50169f7d514d4fe3a4f846faaed03325
I20240503 07:59:29.178213 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 50169f7d514d4fe3a4f846faaed03325 (table rebalancer_extra_table_0 [id=62034a44b5d9412f853d552a90206f04]) successfully deleted
I20240503 07:59:29.220324 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:29.227031 15549 raft_consensus.cc:482] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.227167 15549 raft_consensus.cc:504] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:29.227435 15748 tablet.cc:1722] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.227656 15549 leader_election.cc:288] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:29.227803 15748 tablet.cc:1620] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: MemRowSet was empty: no flush needed.
I20240503 07:59:29.228094 15749 tablet.cc:1722] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.228399 15749 tablet.cc:1620] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:29.228925 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3d64c714fd471b9f055af7cb172a39" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:29.228925 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3d64c714fd471b9f055af7cb172a39" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:29.229045 7282 raft_consensus.cc:2454] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:29.229045 7019 raft_consensus.cc:2454] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:29.229326 9310 leader_election.cc:302] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:29.230064 15828 raft_consensus.cc:2781] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.230125 15828 raft_consensus.cc:482] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.230178 15828 raft_consensus.cc:3037] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.232532 15828 raft_consensus.cc:504] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:29.233971 15758 tablet.cc:1722] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.234300 15758 tablet.cc:1620] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:29.234977 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3d64c714fd471b9f055af7cb172a39" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:29.235123 7019 raft_consensus.cc:3037] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.236001 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=695e494ba1d7474fb2dc6d2794575a22] alter complete (version 2)
I20240503 07:59:29.239562 7019 raft_consensus.cc:2454] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:29.239638 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3d64c714fd471b9f055af7cb172a39" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:29.239338 15828 leader_election.cc:288] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:29.239763 7282 raft_consensus.cc:3037] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.240208 9309 leader_election.cc:302] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:29.240806 7282 raft_consensus.cc:2454] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:29.241506 15828 raft_consensus.cc:2781] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.241786 15828 raft_consensus.cc:686] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:29.242389 15828 consensus_queue.cc:237] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:29.245908 6740 catalog_manager.cc:5484] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:29.339628 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:29.345263 15748 tablet.cc:1722] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:29.345660 15749 tablet.cc:1722] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:29.346649 15758 tablet.cc:1722] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:29.350406 15471 raft_consensus.cc:482] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.350523 15471 raft_consensus.cc:504] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:29.350881 15767 raft_consensus.cc:482] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.350982 15767 raft_consensus.cc:504] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:29.351016 15471 leader_election.cc:288] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:29.351433 15767 leader_election.cc:288] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:29.351819 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab217306da64dd7bb123e7fb890d103" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:29.351948 9374 raft_consensus.cc:2454] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:29.352221 9570 leader_election.cc:302] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:29.352233 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab217306da64dd7bb123e7fb890d103" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:29.352267 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab217306da64dd7bb123e7fb890d103" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:29.352370 9504 raft_consensus.cc:2454] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:29.352370 9635 raft_consensus.cc:2454] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:29.352558 15485 raft_consensus.cc:2781] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.352618 15485 raft_consensus.cc:482] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.352654 15485 raft_consensus.cc:3037] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.353138 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab217306da64dd7bb123e7fb890d103" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:29.353260 9374 raft_consensus.cc:2454] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:29.353495 9439 leader_election.cc:302] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:29.353670 15471 raft_consensus.cc:2781] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.353744 15471 raft_consensus.cc:482] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.353781 15471 raft_consensus.cc:3037] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.353775 15485 raft_consensus.cc:504] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:29.354123 15485 leader_election.cc:288] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:29.354377 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=695e494ba1d7474fb2dc6d2794575a22] alter complete (version 3)
I20240503 07:59:29.354898 15471 raft_consensus.cc:504] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:29.355285 15471 leader_election.cc:288] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:29.355403 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab217306da64dd7bb123e7fb890d103" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:29.355506 9357 raft_consensus.cc:3037] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.355638 9373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab217306da64dd7bb123e7fb890d103" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:29.355765 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab217306da64dd7bb123e7fb890d103" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:29.355924 9504 raft_consensus.cc:2379] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 159177b3caeb434baa98e04136d050b2 in current term 1: Already voted for candidate cb73994515af4880b05383e78368bb4a in this term.
I20240503 07:59:29.356611 9357 raft_consensus.cc:2454] T cab217306da64dd7bb123e7fb890d103 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:29.356869 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab217306da64dd7bb123e7fb890d103" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:29.356921 9570 leader_election.cc:302] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters: cb73994515af4880b05383e78368bb4a
I20240503 07:59:29.357008 9635 raft_consensus.cc:2379] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cb73994515af4880b05383e78368bb4a in current term 1: Already voted for candidate 159177b3caeb434baa98e04136d050b2 in this term.
I20240503 07:59:29.357080 15485 raft_consensus.cc:2781] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.357163 15485 raft_consensus.cc:686] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:29.357296 15485 consensus_queue.cc:237] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:29.357962 9438 leader_election.cc:302] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb73994515af4880b05383e78368bb4a; no voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:29.358204 15471 raft_consensus.cc:2726] T cab217306da64dd7bb123e7fb890d103 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 07:59:29.359517 6740 catalog_manager.cc:5484] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:29.384263 9350 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:29.389283 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:29.393411 7131 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:29.396641 6867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:29.400368 7260 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:29.400856 6998 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:29.404945 9615 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:29.454995 9483 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:29.457473 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:29.457650 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:29.460474 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet d338ae796f6d402d87d2ec1e3e34e416
I20240503 07:59:29.465654 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet d338ae796f6d402d87d2ec1e3e34e416 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:29 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:29.465891 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet d338ae796f6d402d87d2ec1e3e34e416 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:29 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:29.466142 15786 tablet_replica.cc:330] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:29.466230 15843 tablet_replica.cc:330] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:29.466274 15786 raft_consensus.cc:2229] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:29.466413 15843 raft_consensus.cc:2229] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:29.466497 15786 raft_consensus.cc:2258] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.466535 15843 raft_consensus.cc:2258] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.467245 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet d338ae796f6d402d87d2ec1e3e34e416 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:29 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:29.467271 15843 ts_tablet_manager.cc:1875] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:29.468089 15786 ts_tablet_manager.cc:1875] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:29.470047 15786 ts_tablet_manager.cc:1888] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:29.470091 15843 ts_tablet_manager.cc:1888] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:29.470119 15786 log.cc:1196] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d338ae796f6d402d87d2ec1e3e34e416
I20240503 07:59:29.470162 15843 log.cc:1196] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/d338ae796f6d402d87d2ec1e3e34e416
I20240503 07:59:29.470510 15843 ts_tablet_manager.cc:1909] T d338ae796f6d402d87d2ec1e3e34e416 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:29.471906 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet d338ae796f6d402d87d2ec1e3e34e416 (table rebalancer_dynamic_table_new_name [id=695e494ba1d7474fb2dc6d2794575a22]) successfully deleted
I20240503 07:59:29.474412 15786 ts_tablet_manager.cc:1909] T d338ae796f6d402d87d2ec1e3e34e416 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:29.475267 15844 tablet_replica.cc:330] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:29.475315 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet d338ae796f6d402d87d2ec1e3e34e416 (table rebalancer_dynamic_table_new_name [id=695e494ba1d7474fb2dc6d2794575a22]) successfully deleted
I20240503 07:59:29.475507 15844 raft_consensus.cc:2229] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:29.475649 15844 raft_consensus.cc:2258] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.476226 15844 ts_tablet_manager.cc:1875] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:29.478472 15844 ts_tablet_manager.cc:1888] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:29.478543 15844 log.cc:1196] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d338ae796f6d402d87d2ec1e3e34e416
I20240503 07:59:29.478806 15844 ts_tablet_manager.cc:1909] T d338ae796f6d402d87d2ec1e3e34e416 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:29.479799 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet d338ae796f6d402d87d2ec1e3e34e416 (table rebalancer_dynamic_table_new_name [id=695e494ba1d7474fb2dc6d2794575a22]) successfully deleted
I20240503 07:59:29.507721 9635 consensus_queue.cc:237] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:29.509377 9357 raft_consensus.cc:1261] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:29.509562 7282 raft_consensus.cc:1261] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:29.509987 15485 consensus_queue.cc:1035] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:29.510218 15861 consensus_queue.cc:1035] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:29.512117 15485 raft_consensus.cc:2932] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:29.512499 7278 raft_consensus.cc:2932] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:29.516206 9357 raft_consensus.cc:2932] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:29.516343 6740 catalog_manager.cc:5484] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:29.519327 9745 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac0bb7b2c95434aa38f0f0e3718e89c with delete_type TABLET_DATA_TOMBSTONED (TS a372e98a670f49e9adca46ab5c836c4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:29.519703 15868 tablet_replica.cc:330] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:29.519894 15868 raft_consensus.cc:2229] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:29.520072 15868 raft_consensus.cc:2258] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.520593 15868 ts_tablet_manager.cc:1875] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:29.523525 15868 ts_tablet_manager.cc:1888] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:29.523619 15868 log.cc:1196] T 1ac0bb7b2c95434aa38f0f0e3718e89c P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/1ac0bb7b2c95434aa38f0f0e3718e89c
I20240503 07:59:29.524184 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 1ac0bb7b2c95434aa38f0f0e3718e89c (table rebalancer_extra_table_2 [id=bcee57aafa9c44d5a92e47a3e072d94b]) successfully deleted
I20240503 07:59:29.530126 15727 consensus_queue.cc:1035] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.531565 14889 consensus_queue.cc:1035] T 9c7a9734f93e4991a0764c9bb116dac8 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.562414 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:29.568578 9745 tablet_service.cc:1467] Processing CreateTablet for tablet bf7868e9b93c46b7b52a30ffadaaa045 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3b1a4769b71348378d89084c88378a11]), partition=
I20240503 07:59:29.568706 9350 tablet_service.cc:1467] Processing CreateTablet for tablet bf7868e9b93c46b7b52a30ffadaaa045 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3b1a4769b71348378d89084c88378a11]), partition=
I20240503 07:59:29.568872 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7868e9b93c46b7b52a30ffadaaa045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.568929 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7868e9b93c46b7b52a30ffadaaa045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.570629 6867 tablet_service.cc:1467] Processing CreateTablet for tablet bf7868e9b93c46b7b52a30ffadaaa045 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3b1a4769b71348378d89084c88378a11]), partition=
I20240503 07:59:29.570886 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7868e9b93c46b7b52a30ffadaaa045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.571069 15686 tablet_bootstrap.cc:492] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:29.572021 15686 tablet_bootstrap.cc:654] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.573458 15686 tablet_bootstrap.cc:492] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:29.573554 15686 ts_tablet_manager.cc:1367] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:29.573765 15892 tablet_bootstrap.cc:492] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:29.574041 15686 raft_consensus.cc:348] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:29.574177 15686 raft_consensus.cc:374] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.574208 15686 raft_consensus.cc:729] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.574366 15686 consensus_queue.cc:260] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:29.574667 15686 ts_tablet_manager.cc:1398] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.574839 15892 tablet_bootstrap.cc:654] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.575577 15890 tablet_bootstrap.cc:492] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:29.576563 15890 tablet_bootstrap.cc:654] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.577781 15892 tablet_bootstrap.cc:492] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:29.578009 15892 ts_tablet_manager.cc:1367] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:29.578670 15892 raft_consensus.cc:348] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:29.579550 15892 raft_consensus.cc:374] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.579667 15892 raft_consensus.cc:729] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.579810 15892 consensus_queue.cc:260] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:29.580117 15892 ts_tablet_manager.cc:1398] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:29.582765 15890 tablet_bootstrap.cc:492] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:29.582880 15890 ts_tablet_manager.cc:1367] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:29.583400 15890 raft_consensus.cc:348] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:29.583518 15890 raft_consensus.cc:374] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.583550 15890 raft_consensus.cc:729] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.583681 15890 consensus_queue.cc:260] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:29.583993 15890 ts_tablet_manager.cc:1398] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.715945 15673 raft_consensus.cc:482] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.716089 15673 raft_consensus.cc:504] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:29.716665 15673 leader_election.cc:288] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:29.717115 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7868e9b93c46b7b52a30ffadaaa045" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:29.717144 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7868e9b93c46b7b52a30ffadaaa045" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:29.717270 9767 raft_consensus.cc:2454] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:29.717274 9357 raft_consensus.cc:2454] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:29.717589 6825 leader_election.cc:302] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:29.717798 15673 raft_consensus.cc:2781] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.717862 15673 raft_consensus.cc:482] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.717900 15673 raft_consensus.cc:3037] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.718885 15673 raft_consensus.cc:504] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:29.719213 15673 leader_election.cc:288] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:29.719504 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7868e9b93c46b7b52a30ffadaaa045" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:29.719537 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7868e9b93c46b7b52a30ffadaaa045" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:29.719627 9767 raft_consensus.cc:3037] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.719643 9357 raft_consensus.cc:3037] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.720400 9767 raft_consensus.cc:2454] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:29.720552 9357 raft_consensus.cc:2454] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:29.720680 6825 leader_election.cc:302] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:29.720846 15673 raft_consensus.cc:2781] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.720949 15673 raft_consensus.cc:686] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:29.721096 15673 consensus_queue.cc:237] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:29.723428 6739 catalog_manager.cc:5484] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:29.725598 15828 consensus_queue.cc:1035] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.730978 15549 consensus_queue.cc:1035] T bc3d64c714fd471b9f055af7cb172a39 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.751314 9615 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:29.752830 7131 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:29.754300 15861 consensus_queue.cc:1035] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.760951 6998 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:29.764744 15767 consensus_queue.cc:1035] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.774894 7260 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:29.776037 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:29.777096 9350 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:29.784466 9483 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:29.796413 6867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:29.829852 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:29.836186 9374 raft_consensus.cc:1261] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:29.836681 15506 consensus_queue.cc:1035] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.836755 9767 raft_consensus.cc:1261] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:29.837280 15506 consensus_queue.cc:1035] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.841897 6739 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet bf7868e9b93c46b7b52a30ffadaaa045 (table rebalancer_dynamic_table [id=3b1a4769b71348378d89084c88378a11]). Expected version 1 got 0
I20240503 07:59:29.842743 6745 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet bf7868e9b93c46b7b52a30ffadaaa045 (table rebalancer_dynamic_table [id=3b1a4769b71348378d89084c88378a11]). Expected version 1 got 0
I20240503 07:59:29.843189 15758 tablet.cc:1722] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:29.843472 15758 tablet.cc:1620] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:29.846074 6745 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet bf7868e9b93c46b7b52a30ffadaaa045 (table rebalancer_dynamic_table [id=3b1a4769b71348378d89084c88378a11]). Expected version 1 got 0
I20240503 07:59:29.849017 15941 tablet.cc:1722] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:29.849498 15941 tablet.cc:1620] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:29.850047 15946 tablet.cc:1722] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:29.850132 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=3b1a4769b71348378d89084c88378a11] alter complete (version 1)
I20240503 07:59:29.850394 15946 tablet.cc:1620] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:29.851706 6745 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet bf7868e9b93c46b7b52a30ffadaaa045 (table rebalancer_dynamic_table [id=3b1a4769b71348378d89084c88378a11]). Expected version 1 got 0
I20240503 07:59:29.868913 7278 consensus_queue.cc:237] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.870323 6745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:29.870728 7020 raft_consensus.cc:1261] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:29.871125 9766 raft_consensus.cc:1261] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:29.871111 14839 consensus_queue.cc:1035] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:29.871493 14839 consensus_queue.cc:1035] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:29.873113 14440 raft_consensus.cc:2932] T 4737c826ebb64177a718d83b939c3e7b P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:29.874071 9766 raft_consensus.cc:2932] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:29.874141 7020 raft_consensus.cc:2932] T 4737c826ebb64177a718d83b939c3e7b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:29.879261 6746 catalog_manager.cc:5484] T 4737c826ebb64177a718d83b939c3e7b P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:29.882093 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet 4737c826ebb64177a718d83b939c3e7b with delete_type TABLET_DATA_TOMBSTONED (TS 3befbb2d206f4be89d3e07f38837374b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:29.882508 15843 tablet_replica.cc:330] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:29.882686 15843 raft_consensus.cc:2229] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:29.882835 15843 raft_consensus.cc:2258] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.883293 15843 ts_tablet_manager.cc:1875] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:29.885680 15843 ts_tablet_manager.cc:1888] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:29.885746 15843 log.cc:1196] T 4737c826ebb64177a718d83b939c3e7b P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4737c826ebb64177a718d83b939c3e7b
I20240503 07:59:29.886234 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 4737c826ebb64177a718d83b939c3e7b (table rebalancer_extra_table_4 [id=9b4298d7d24e485d91d7ae96bac125e2]) successfully deleted
I20240503 07:59:29.888943 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 25979eb7a4fe48939d0585bf27468918 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2bf48564f74e4877a3ef42f1697337d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.889075 9615 tablet_service.cc:1467] Processing CreateTablet for tablet b1a24370303d4b16abf34d0449d9f2e8 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2bf48564f74e4877a3ef42f1697337d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.889278 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25979eb7a4fe48939d0585bf27468918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.889223 6999 tablet_service.cc:1467] Processing CreateTablet for tablet b1a24370303d4b16abf34d0449d9f2e8 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2bf48564f74e4877a3ef42f1697337d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.889251 7131 tablet_service.cc:1467] Processing CreateTablet for tablet b1a24370303d4b16abf34d0449d9f2e8 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2bf48564f74e4877a3ef42f1697337d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.889367 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a24370303d4b16abf34d0449d9f2e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.889467 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a24370303d4b16abf34d0449d9f2e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.889524 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a24370303d4b16abf34d0449d9f2e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.892143 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 25979eb7a4fe48939d0585bf27468918 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2bf48564f74e4877a3ef42f1697337d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.892138 9616 tablet_service.cc:1467] Processing CreateTablet for tablet e42b9f82600e4ff1b03bf2d804cb65de (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2bf48564f74e4877a3ef42f1697337d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.892213 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 25979eb7a4fe48939d0585bf27468918 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2bf48564f74e4877a3ef42f1697337d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.892391 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25979eb7a4fe48939d0585bf27468918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.892391 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e42b9f82600e4ff1b03bf2d804cb65de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.892482 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25979eb7a4fe48939d0585bf27468918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.894004 15686 tablet_bootstrap.cc:492] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:29.894191 7260 tablet_service.cc:1467] Processing CreateTablet for tablet e42b9f82600e4ff1b03bf2d804cb65de (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2bf48564f74e4877a3ef42f1697337d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.894485 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e42b9f82600e4ff1b03bf2d804cb65de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.894657 15962 tablet_bootstrap.cc:492] T b1a24370303d4b16abf34d0449d9f2e8 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:29.895085 15686 tablet_bootstrap.cc:654] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.895653 15962 tablet_bootstrap.cc:654] T b1a24370303d4b16abf34d0449d9f2e8 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.896073 15961 tablet_bootstrap.cc:492] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:29.896739 15686 tablet_bootstrap.cc:492] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:29.896847 15686 ts_tablet_manager.cc:1367] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:29.897315 15686 raft_consensus.cc:348] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.897428 15686 raft_consensus.cc:374] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.897456 15686 raft_consensus.cc:729] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.897634 15686 consensus_queue.cc:260] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.898110 15961 tablet_bootstrap.cc:654] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.899124 15686 ts_tablet_manager.cc:1398] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:29.899394 15966 tablet_bootstrap.cc:492] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:29.899899 15962 tablet_bootstrap.cc:492] T b1a24370303d4b16abf34d0449d9f2e8 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:29.900012 15962 ts_tablet_manager.cc:1367] T b1a24370303d4b16abf34d0449d9f2e8 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:29.900193 15961 tablet_bootstrap.cc:492] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:29.900300 15961 ts_tablet_manager.cc:1367] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 07:59:29.900511 15962 raft_consensus.cc:348] T b1a24370303d4b16abf34d0449d9f2e8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:29.900640 15962 raft_consensus.cc:374] T b1a24370303d4b16abf34d0449d9f2e8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.900691 15962 raft_consensus.cc:729] T b1a24370303d4b16abf34d0449d9f2e8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.900758 15961 raft_consensus.cc:348] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:29.900861 15961 raft_consensus.cc:374] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.900897 15961 raft_consensus.cc:729] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.900830 15962 consensus_queue.cc:260] T b1a24370303d4b16abf34d0449d9f2e8 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:29.901002 15961 consensus_queue.cc:260] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:29.901281 15962 ts_tablet_manager.cc:1398] T b1a24370303d4b16abf34d0449d9f2e8 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.901324 15961 ts_tablet_manager.cc:1398] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:29.902304 15966 tablet_bootstrap.cc:654] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.905122 15966 tablet_bootstrap.cc:492] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:29.905220 15966 ts_tablet_manager.cc:1367] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:29.905612 15966 raft_consensus.cc:348] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:29.905723 15966 raft_consensus.cc:374] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.905756 15966 raft_consensus.cc:729] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.905881 15966 consensus_queue.cc:260] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:29.906191 15966 ts_tablet_manager.cc:1398] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:29.908128 15961 tablet_bootstrap.cc:492] T e42b9f82600e4ff1b03bf2d804cb65de P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:29.910037 15890 tablet_bootstrap.cc:492] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:29.910671 15961 tablet_bootstrap.cc:654] T e42b9f82600e4ff1b03bf2d804cb65de P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.911053 15890 tablet_bootstrap.cc:654] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.912699 15890 tablet_bootstrap.cc:492] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:29.912794 15890 ts_tablet_manager.cc:1367] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:29.913195 15890 raft_consensus.cc:348] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.913283 15890 raft_consensus.cc:374] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.913311 15890 raft_consensus.cc:729] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.913430 15890 consensus_queue.cc:260] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.913726 15890 ts_tablet_manager.cc:1398] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:29.914451 7131 tablet_service.cc:1467] Processing CreateTablet for tablet e42b9f82600e4ff1b03bf2d804cb65de (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2bf48564f74e4877a3ef42f1697337d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.914577 15963 tablet_bootstrap.cc:492] T b1a24370303d4b16abf34d0449d9f2e8 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:29.915587 15963 tablet_bootstrap.cc:654] T b1a24370303d4b16abf34d0449d9f2e8 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.915913 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e42b9f82600e4ff1b03bf2d804cb65de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.918251 15962 tablet_bootstrap.cc:492] T e42b9f82600e4ff1b03bf2d804cb65de P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:29.919287 15962 tablet_bootstrap.cc:654] T e42b9f82600e4ff1b03bf2d804cb65de P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.920537 15962 tablet_bootstrap.cc:492] T e42b9f82600e4ff1b03bf2d804cb65de P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:29.920645 15962 ts_tablet_manager.cc:1367] T e42b9f82600e4ff1b03bf2d804cb65de P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:29.921080 15962 raft_consensus.cc:348] T e42b9f82600e4ff1b03bf2d804cb65de P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:29.921197 15962 raft_consensus.cc:374] T e42b9f82600e4ff1b03bf2d804cb65de P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.921229 15962 raft_consensus.cc:729] T e42b9f82600e4ff1b03bf2d804cb65de P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.921339 15962 consensus_queue.cc:260] T e42b9f82600e4ff1b03bf2d804cb65de P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:29.922211 15962 ts_tablet_manager.cc:1398] T e42b9f82600e4ff1b03bf2d804cb65de P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:29.922721 15961 tablet_bootstrap.cc:492] T e42b9f82600e4ff1b03bf2d804cb65de P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:29.922821 15961 ts_tablet_manager.cc:1367] T e42b9f82600e4ff1b03bf2d804cb65de P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240503 07:59:29.923367 15961 raft_consensus.cc:348] T e42b9f82600e4ff1b03bf2d804cb65de P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:29.923465 15961 raft_consensus.cc:374] T e42b9f82600e4ff1b03bf2d804cb65de P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.923493 15961 raft_consensus.cc:729] T e42b9f82600e4ff1b03bf2d804cb65de P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.923611 15961 consensus_queue.cc:260] T e42b9f82600e4ff1b03bf2d804cb65de P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:29.923878 15961 ts_tablet_manager.cc:1398] T e42b9f82600e4ff1b03bf2d804cb65de P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:29.926653 15963 tablet_bootstrap.cc:492] T b1a24370303d4b16abf34d0449d9f2e8 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:29.926774 15963 ts_tablet_manager.cc:1367] T b1a24370303d4b16abf34d0449d9f2e8 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:29.927280 15963 raft_consensus.cc:348] T b1a24370303d4b16abf34d0449d9f2e8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:29.927387 15963 raft_consensus.cc:374] T b1a24370303d4b16abf34d0449d9f2e8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.927415 15963 raft_consensus.cc:729] T b1a24370303d4b16abf34d0449d9f2e8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.927562 15963 consensus_queue.cc:260] T b1a24370303d4b16abf34d0449d9f2e8 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:29.927839 15963 ts_tablet_manager.cc:1398] T b1a24370303d4b16abf34d0449d9f2e8 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.928030 15963 tablet_bootstrap.cc:492] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:29.928951 15963 tablet_bootstrap.cc:654] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.930886 15963 tablet_bootstrap.cc:492] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:29.930990 15963 ts_tablet_manager.cc:1367] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:29.931529 15963 raft_consensus.cc:348] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.931648 15963 raft_consensus.cc:374] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.931700 15963 raft_consensus.cc:729] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.931937 15963 consensus_queue.cc:260] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.932264 15963 ts_tablet_manager.cc:1398] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.941644 15549 raft_consensus.cc:482] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.941766 15549 raft_consensus.cc:504] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.942281 15549 leader_election.cc:288] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:29.942718 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25979eb7a4fe48939d0585bf27468918" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:29.942870 9766 raft_consensus.cc:2454] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:29.943324 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25979eb7a4fe48939d0585bf27468918" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:29.943372 14839 raft_consensus.cc:482] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.943462 7020 raft_consensus.cc:2454] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:29.943477 9310 leader_election.cc:302] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:29.943476 14839 raft_consensus.cc:504] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:29.943643 15549 raft_consensus.cc:2781] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.943691 15549 raft_consensus.cc:482] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.943720 15549 raft_consensus.cc:3037] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.943866 14839 leader_election.cc:288] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:29.944406 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e42b9f82600e4ff1b03bf2d804cb65de" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:29.944540 9635 raft_consensus.cc:2454] T e42b9f82600e4ff1b03bf2d804cb65de P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:29.944870 7215 leader_election.cc:302] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:29.944891 15549 raft_consensus.cc:504] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.945029 14839 raft_consensus.cc:2781] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.945093 14839 raft_consensus.cc:482] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.945132 14839 raft_consensus.cc:3037] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.945232 15549 leader_election.cc:288] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:29.945884 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e42b9f82600e4ff1b03bf2d804cb65de" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:29.946014 7149 raft_consensus.cc:2454] T e42b9f82600e4ff1b03bf2d804cb65de P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:29.946276 14839 raft_consensus.cc:504] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:29.946329 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25979eb7a4fe48939d0585bf27468918" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:29.946485 9766 raft_consensus.cc:3037] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.946651 14839 leader_election.cc:288] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:29.947149 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e42b9f82600e4ff1b03bf2d804cb65de" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:29.947253 7149 raft_consensus.cc:3037] T e42b9f82600e4ff1b03bf2d804cb65de P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.947295 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e42b9f82600e4ff1b03bf2d804cb65de" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:29.947396 9635 raft_consensus.cc:3037] T e42b9f82600e4ff1b03bf2d804cb65de P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.947414 9766 raft_consensus.cc:2454] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:29.947683 9310 leader_election.cc:302] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:29.947832 15549 raft_consensus.cc:2781] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.947904 15549 raft_consensus.cc:686] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:29.948040 15549 consensus_queue.cc:237] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:29.948254 7149 raft_consensus.cc:2454] T e42b9f82600e4ff1b03bf2d804cb65de P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:29.948326 9635 raft_consensus.cc:2454] T e42b9f82600e4ff1b03bf2d804cb65de P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:29.948630 7215 leader_election.cc:302] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:29.948875 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25979eb7a4fe48939d0585bf27468918" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:29.949000 7019 raft_consensus.cc:3037] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.949880 7019 raft_consensus.cc:2454] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:29.950184 6739 catalog_manager.cc:5484] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:29.950634 14839 raft_consensus.cc:2781] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.950723 14839 raft_consensus.cc:686] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:29.950863 14839 consensus_queue.cc:237] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:29.953759 6739 catalog_manager.cc:5484] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:29.955624 6745 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:29.960983 15941 tablet.cc:1722] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.961320 15941 tablet.cc:1620] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:29.961758 15758 tablet.cc:1722] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.962059 15758 tablet.cc:1620] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:29.962858 15946 tablet.cc:1722] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.963160 15946 tablet.cc:1620] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:29.969017 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=3b1a4769b71348378d89084c88378a11] alter complete (version 2)
I20240503 07:59:30.032500 15485 raft_consensus.cc:482] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.032646 15485 raft_consensus.cc:504] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:30.033098 15485 leader_election.cc:288] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:30.033560 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a24370303d4b16abf34d0449d9f2e8" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:30.033707 7020 raft_consensus.cc:2454] T b1a24370303d4b16abf34d0449d9f2e8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:30.033753 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a24370303d4b16abf34d0449d9f2e8" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:30.033891 7149 raft_consensus.cc:2454] T b1a24370303d4b16abf34d0449d9f2e8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:30.034147 9571 leader_election.cc:302] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:30.034405 15485 raft_consensus.cc:2781] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.034458 15485 raft_consensus.cc:482] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.034490 15485 raft_consensus.cc:3037] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.035694 15485 raft_consensus.cc:504] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:30.036072 15485 leader_election.cc:288] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:30.036621 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a24370303d4b16abf34d0449d9f2e8" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:30.036749 7149 raft_consensus.cc:3037] T b1a24370303d4b16abf34d0449d9f2e8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.036724 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a24370303d4b16abf34d0449d9f2e8" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:30.036844 7020 raft_consensus.cc:3037] T b1a24370303d4b16abf34d0449d9f2e8 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.037918 7149 raft_consensus.cc:2454] T b1a24370303d4b16abf34d0449d9f2e8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:30.037920 7020 raft_consensus.cc:2454] T b1a24370303d4b16abf34d0449d9f2e8 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:30.038266 9572 leader_election.cc:302] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2; no voters:
I20240503 07:59:30.038488 15485 raft_consensus.cc:2781] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.038556 15485 raft_consensus.cc:686] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:30.038727 15485 consensus_queue.cc:237] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:30.041868 6741 catalog_manager.cc:5484] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:30.071795 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:30.073802 7131 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:30.082080 6998 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:30.084384 15941 tablet.cc:1722] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.086087 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:30.090690 9350 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:30.090729 15946 tablet.cc:1722] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.093430 9616 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:30.093567 15758 tablet.cc:1722] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.095965 7260 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:30.099267 9483 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:30.110548 6867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:30.110929 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=3b1a4769b71348378d89084c88378a11] alter complete (version 3)
I20240503 07:59:30.189883 7278 consensus_queue.cc:237] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:30.191854 9374 raft_consensus.cc:1261] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:30.191956 6886 raft_consensus.cc:1261] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:30.192384 14839 consensus_queue.cc:1035] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:30.192591 14440 consensus_queue.cc:1035] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:30.194900 14839 raft_consensus.cc:2932] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:30.196873 9374 raft_consensus.cc:2932] T 721c34a8b6a348e5890d462ba7da938c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:30.198220 6746 catalog_manager.cc:5484] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:30.201881 7131 tablet_service.cc:1514] Processing DeleteTablet for tablet 721c34a8b6a348e5890d462ba7da938c with delete_type TABLET_DATA_TOMBSTONED (TS 00a3f8e159c64327832d79fb32cdcca8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:30.202594 16047 tablet_replica.cc:330] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:30.202775 16047 raft_consensus.cc:2229] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.203125 16047 raft_consensus.cc:2258] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.203780 16047 ts_tablet_manager.cc:1875] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:30.206277 16047 ts_tablet_manager.cc:1888] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:30.206442 16047 log.cc:1196] T 721c34a8b6a348e5890d462ba7da938c P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/721c34a8b6a348e5890d462ba7da938c
I20240503 07:59:30.207010 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 721c34a8b6a348e5890d462ba7da938c (table rebalance_test_table_19 [id=071fb84151514ba094acc7ff3c1b24e7]) successfully deleted
I20240503 07:59:30.208055 6886 raft_consensus.cc:2932] T 721c34a8b6a348e5890d462ba7da938c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:30.214550 6739 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:30.214761 6739 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:30.217415 6739 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet bf7868e9b93c46b7b52a30ffadaaa045
I20240503 07:59:30.218254 9745 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7868e9b93c46b7b52a30ffadaaa045 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:30.218394 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7868e9b93c46b7b52a30ffadaaa045 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:30.218709 16056 tablet_replica.cc:330] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:30.218879 16056 raft_consensus.cc:2229] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.219013 16056 raft_consensus.cc:2258] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.219367 16057 tablet_replica.cc:330] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:30.219539 16057 raft_consensus.cc:2229] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:30.219799 16057 raft_consensus.cc:2258] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.220196 16056 ts_tablet_manager.cc:1875] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.221944 16057 ts_tablet_manager.cc:1875] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.222744 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7868e9b93c46b7b52a30ffadaaa045 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:30.222867 16056 ts_tablet_manager.cc:1888] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.222965 16056 log.cc:1196] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/bf7868e9b93c46b7b52a30ffadaaa045
I20240503 07:59:30.222978 15843 tablet_replica.cc:330] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:30.223125 15843 raft_consensus.cc:2229] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.223249 16056 ts_tablet_manager.cc:1909] T bf7868e9b93c46b7b52a30ffadaaa045 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:30.223258 15843 raft_consensus.cc:2258] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.223966 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet bf7868e9b93c46b7b52a30ffadaaa045 (table rebalancer_dynamic_table_new_name [id=3b1a4769b71348378d89084c88378a11]) successfully deleted
I20240503 07:59:30.226326 16057 ts_tablet_manager.cc:1888] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.226440 16057 log.cc:1196] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/bf7868e9b93c46b7b52a30ffadaaa045
I20240503 07:59:30.226820 16057 ts_tablet_manager.cc:1909] T bf7868e9b93c46b7b52a30ffadaaa045 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:30.227566 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet bf7868e9b93c46b7b52a30ffadaaa045 (table rebalancer_dynamic_table_new_name [id=3b1a4769b71348378d89084c88378a11]) successfully deleted
I20240503 07:59:30.229822 15843 ts_tablet_manager.cc:1875] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.231968 15843 ts_tablet_manager.cc:1888] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.232049 15843 log.cc:1196] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/bf7868e9b93c46b7b52a30ffadaaa045
I20240503 07:59:30.232295 15843 ts_tablet_manager.cc:1909] T bf7868e9b93c46b7b52a30ffadaaa045 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:30.233207 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet bf7868e9b93c46b7b52a30ffadaaa045 (table rebalancer_dynamic_table_new_name [id=3b1a4769b71348378d89084c88378a11]) successfully deleted
I20240503 07:59:30.320973 7260 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:30.323036 7131 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:30.323714 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:30.327236 6998 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:30.327522 9350 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:30.330521 9616 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:30.331419 14440 consensus_queue.cc:1035] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.334514 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 0b6e89e693fd40d28d818bc97c0e546a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6b7cebfa760b461385cc83a4b43a4de6]), partition=
I20240503 07:59:30.334687 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 0b6e89e693fd40d28d818bc97c0e546a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6b7cebfa760b461385cc83a4b43a4de6]), partition=
I20240503 07:59:30.334841 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b6e89e693fd40d28d818bc97c0e546a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.334949 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b6e89e693fd40d28d818bc97c0e546a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.337049 15686 tablet_bootstrap.cc:492] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:30.337302 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 0b6e89e693fd40d28d818bc97c0e546a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6b7cebfa760b461385cc83a4b43a4de6]), partition=
I20240503 07:59:30.337555 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b6e89e693fd40d28d818bc97c0e546a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.338027 15686 tablet_bootstrap.cc:654] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.341081 15686 tablet_bootstrap.cc:492] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:30.341205 15686 ts_tablet_manager.cc:1367] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 07:59:30.341322 16100 tablet_bootstrap.cc:492] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:30.341624 16104 tablet_bootstrap.cc:492] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:30.341701 15686 raft_consensus.cc:348] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:30.341820 15686 raft_consensus.cc:374] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.341850 15686 raft_consensus.cc:729] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.341972 15686 consensus_queue.cc:260] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:30.342275 15686 ts_tablet_manager.cc:1398] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.343899 16100 tablet_bootstrap.cc:654] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.344952 16104 tablet_bootstrap.cc:654] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.346493 16104 tablet_bootstrap.cc:492] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:30.346637 16104 ts_tablet_manager.cc:1367] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:30.346771 16100 tablet_bootstrap.cc:492] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:30.346923 16100 ts_tablet_manager.cc:1367] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:30.347190 16104 raft_consensus.cc:348] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:30.347401 16104 raft_consensus.cc:374] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.347482 16104 raft_consensus.cc:729] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.347424 16100 raft_consensus.cc:348] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:30.347661 16100 raft_consensus.cc:374] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.347733 16100 raft_consensus.cc:729] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.347836 16104 consensus_queue.cc:260] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:30.347920 16100 consensus_queue.cc:260] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:30.348070 15506 raft_consensus.cc:482] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.348126 15506 raft_consensus.cc:504] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:30.348265 16104 ts_tablet_manager.cc:1398] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:30.348517 15506 leader_election.cc:288] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:30.349666 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b6e89e693fd40d28d818bc97c0e546a" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:30.349790 9357 raft_consensus.cc:2454] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:30.350139 6823 leader_election.cc:302] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:30.350399 15506 raft_consensus.cc:2781] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.350445 15506 raft_consensus.cc:482] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.350476 15506 raft_consensus.cc:3037] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.350637 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b6e89e693fd40d28d818bc97c0e546a" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:30.350777 9504 raft_consensus.cc:2454] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:30.351475 14440 consensus_queue.cc:1035] T e42b9f82600e4ff1b03bf2d804cb65de P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.351559 15506 raft_consensus.cc:504] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:30.351894 15506 leader_election.cc:288] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:30.352685 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b6e89e693fd40d28d818bc97c0e546a" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:30.352809 9505 raft_consensus.cc:3037] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.353919 9505 raft_consensus.cc:2454] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:30.355041 16100 ts_tablet_manager.cc:1398] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:30.356431 6825 leader_election.cc:302] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:30.356947 15993 raft_consensus.cc:2781] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.357043 15993 raft_consensus.cc:686] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:30.357175 15993 consensus_queue.cc:237] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:30.360116 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b6e89e693fd40d28d818bc97c0e546a" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:30.360258 9357 raft_consensus.cc:3037] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.361441 9357 raft_consensus.cc:2454] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:30.366765 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:30.368341 6744 catalog_manager.cc:5484] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:30.370496 6867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:30.386025 9483 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:30.461413 6886 consensus_queue.cc:237] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:30.461759 15767 consensus_queue.cc:1035] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.462828 9635 raft_consensus.cc:1261] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:30.463208 15673 consensus_queue.cc:1035] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:30.463758 9767 raft_consensus.cc:1261] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:30.464128 15673 consensus_queue.cc:1035] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:30.465939 15485 consensus_queue.cc:1035] T b1a24370303d4b16abf34d0449d9f2e8 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.466389 15993 raft_consensus.cc:2932] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:30.469904 6744 catalog_manager.cc:5484] T cf8e0a0da8b941e28e797de8f8b7a3f4 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index 2 to 3, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:30.470811 9635 raft_consensus.cc:2932] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:30.473402 15549 consensus_queue.cc:1035] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.473953 9767 raft_consensus.cc:2932] T cf8e0a0da8b941e28e797de8f8b7a3f4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:30.477368 6744 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:30.477843 7131 tablet_service.cc:1514] Processing DeleteTablet for tablet cf8e0a0da8b941e28e797de8f8b7a3f4 with delete_type TABLET_DATA_TOMBSTONED (TS 00a3f8e159c64327832d79fb32cdcca8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:30.478098 16047 tablet_replica.cc:330] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:30.478260 16047 raft_consensus.cc:2229] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.478427 16047 raft_consensus.cc:2258] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.478948 16047 ts_tablet_manager.cc:1875] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:30.481431 16047 ts_tablet_manager.cc:1888] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:30.481487 16047 log.cc:1196] T cf8e0a0da8b941e28e797de8f8b7a3f4 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/cf8e0a0da8b941e28e797de8f8b7a3f4
I20240503 07:59:30.482136 15829 consensus_queue.cc:1035] T 25979eb7a4fe48939d0585bf27468918 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.483286 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet cf8e0a0da8b941e28e797de8f8b7a3f4 (table rebalancer_extra_table_10 [id=fa3c77fefb014f88b3cad9f4ea921534]) successfully deleted
I20240503 07:59:30.484055 9357 raft_consensus.cc:1261] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:30.484405 9505 raft_consensus.cc:1261] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:30.484721 15993 consensus_queue.cc:1035] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.484928 15673 consensus_queue.cc:1035] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.488418 6744 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 0b6e89e693fd40d28d818bc97c0e546a (table rebalancer_dynamic_table [id=6b7cebfa760b461385cc83a4b43a4de6]). Expected version 1 got 0
I20240503 07:59:30.488983 15941 tablet.cc:1722] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:30.489343 15941 tablet.cc:1620] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:30.491343 6744 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet 0b6e89e693fd40d28d818bc97c0e546a (table rebalancer_dynamic_table [id=6b7cebfa760b461385cc83a4b43a4de6]). Expected version 1 got 0
I20240503 07:59:30.495780 15758 tablet.cc:1722] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:30.496145 15758 tablet.cc:1620] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:30.497736 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=6b7cebfa760b461385cc83a4b43a4de6] alter complete (version 1)
I20240503 07:59:30.501221 16150 tablet.cc:1722] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:30.501631 16150 tablet.cc:1620] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:30.579051 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:30.589615 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa18637596d4a39aff9a4e4a261228d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c023c0d7842647458f0b751b3ef0170d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.589615 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef6b8a139ca4be5bc496b6678b6c1a2 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c023c0d7842647458f0b751b3ef0170d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.589888 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef6b8a139ca4be5bc496b6678b6c1a2 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c023c0d7842647458f0b751b3ef0170d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.589964 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa18637596d4a39aff9a4e4a261228d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.589960 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef6b8a139ca4be5bc496b6678b6c1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.590101 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef6b8a139ca4be5bc496b6678b6c1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.590050 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa18637596d4a39aff9a4e4a261228d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c023c0d7842647458f0b751b3ef0170d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.590307 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa18637596d4a39aff9a4e4a261228d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.590672 9483 tablet_service.cc:1467] Processing CreateTablet for tablet cb6d879affda4232af790bf7d38d5558 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c023c0d7842647458f0b751b3ef0170d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.590922 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6d879affda4232af790bf7d38d5558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.592933 7131 tablet_service.cc:1467] Processing CreateTablet for tablet cb6d879affda4232af790bf7d38d5558 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c023c0d7842647458f0b751b3ef0170d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.593116 16104 tablet_bootstrap.cc:492] T cb6d879affda4232af790bf7d38d5558 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:30.593212 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6d879affda4232af790bf7d38d5558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.593449 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa18637596d4a39aff9a4e4a261228d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c023c0d7842647458f0b751b3ef0170d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.593716 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa18637596d4a39aff9a4e4a261228d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.594173 16104 tablet_bootstrap.cc:654] T cb6d879affda4232af790bf7d38d5558 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.595840 16183 tablet_bootstrap.cc:492] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:30.595821 7260 tablet_service.cc:1467] Processing CreateTablet for tablet cb6d879affda4232af790bf7d38d5558 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c023c0d7842647458f0b751b3ef0170d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.595975 16100 tablet_bootstrap.cc:492] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:30.596092 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6d879affda4232af790bf7d38d5558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.596935 16100 tablet_bootstrap.cc:654] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.597148 16183 tablet_bootstrap.cc:654] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.597899 15686 tablet_bootstrap.cc:492] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:30.599926 15686 tablet_bootstrap.cc:654] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.599974 16183 tablet_bootstrap.cc:492] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:30.600102 16183 ts_tablet_manager.cc:1367] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:30.600256 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef6b8a139ca4be5bc496b6678b6c1a2 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c023c0d7842647458f0b751b3ef0170d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.600526 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef6b8a139ca4be5bc496b6678b6c1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.600581 16183 raft_consensus.cc:348] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:30.600692 16183 raft_consensus.cc:374] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.600718 16183 raft_consensus.cc:729] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.600833 16183 consensus_queue.cc:260] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:30.601122 16183 ts_tablet_manager.cc:1398] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.602610 6743 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:30.603562 16185 tablet_bootstrap.cc:492] T cb6d879affda4232af790bf7d38d5558 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:30.604588 16185 tablet_bootstrap.cc:654] T cb6d879affda4232af790bf7d38d5558 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.605278 16181 tablet_bootstrap.cc:492] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:30.606583 16181 tablet_bootstrap.cc:654] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.607745 16188 tablet_bootstrap.cc:492] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:30.608832 16188 tablet_bootstrap.cc:654] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.609504 16104 tablet_bootstrap.cc:492] T cb6d879affda4232af790bf7d38d5558 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:30.609601 16104 ts_tablet_manager.cc:1367] T cb6d879affda4232af790bf7d38d5558 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 07:59:30.610123 16104 raft_consensus.cc:348] T cb6d879affda4232af790bf7d38d5558 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:30.610240 16104 raft_consensus.cc:374] T cb6d879affda4232af790bf7d38d5558 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.610273 16104 raft_consensus.cc:729] T cb6d879affda4232af790bf7d38d5558 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.610410 16104 consensus_queue.cc:260] T cb6d879affda4232af790bf7d38d5558 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:30.610707 16104 ts_tablet_manager.cc:1398] T cb6d879affda4232af790bf7d38d5558 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.611876 15686 tablet_bootstrap.cc:492] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:30.611977 15686 ts_tablet_manager.cc:1367] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240503 07:59:30.612388 15941 tablet.cc:1722] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:30.612469 15686 raft_consensus.cc:348] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:30.612568 15686 raft_consensus.cc:374] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.612596 15686 raft_consensus.cc:729] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.612730 15941 tablet.cc:1620] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:30.612738 15686 consensus_queue.cc:260] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:30.613034 15686 ts_tablet_manager.cc:1398] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.613655 16150 tablet.cc:1722] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:30.613947 16150 tablet.cc:1620] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:30.620509 16181 tablet_bootstrap.cc:492] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:30.620714 16181 ts_tablet_manager.cc:1367] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240503 07:59:30.620788 6743 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=6b7cebfa760b461385cc83a4b43a4de6] alter complete (version 2)
I20240503 07:59:30.621306 16181 raft_consensus.cc:348] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:30.621510 16181 raft_consensus.cc:374] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.621649 16181 raft_consensus.cc:729] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.621846 16181 consensus_queue.cc:260] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:30.623382 16181 ts_tablet_manager.cc:1398] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:30.623617 16181 tablet_bootstrap.cc:492] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:30.624953 15758 tablet.cc:1722] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:30.625305 15758 tablet.cc:1620] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:30.626175 16181 tablet_bootstrap.cc:654] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.630954 16100 tablet_bootstrap.cc:492] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:30.631125 16100 ts_tablet_manager.cc:1367] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240503 07:59:30.631453 9616 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:30.631640 16100 raft_consensus.cc:348] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:30.631759 16100 raft_consensus.cc:374] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.631793 16100 raft_consensus.cc:729] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.631906 16100 consensus_queue.cc:260] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:30.632226 16100 ts_tablet_manager.cc:1398] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:30.632390 16100 tablet_bootstrap.cc:492] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:30.633466 16100 tablet_bootstrap.cc:654] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.640424 16185 tablet_bootstrap.cc:492] T cb6d879affda4232af790bf7d38d5558 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:30.640611 16185 ts_tablet_manager.cc:1367] T cb6d879affda4232af790bf7d38d5558 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240503 07:59:30.641254 16185 raft_consensus.cc:348] T cb6d879affda4232af790bf7d38d5558 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:30.641628 16185 raft_consensus.cc:374] T cb6d879affda4232af790bf7d38d5558 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.641703 16185 raft_consensus.cc:729] T cb6d879affda4232af790bf7d38d5558 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.641901 16185 consensus_queue.cc:260] T cb6d879affda4232af790bf7d38d5558 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:30.642094 9350 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:30.643287 16185 ts_tablet_manager.cc:1398] T cb6d879affda4232af790bf7d38d5558 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:30.643944 16100 tablet_bootstrap.cc:492] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:30.644060 16100 ts_tablet_manager.cc:1367] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:30.644562 16100 raft_consensus.cc:348] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:30.644673 16100 raft_consensus.cc:374] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.644699 16100 raft_consensus.cc:729] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.644817 16100 consensus_queue.cc:260] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:30.645136 16100 ts_tablet_manager.cc:1398] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:30.651029 7131 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:30.664086 6867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:30.670166 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:30.675019 6998 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:30.685209 16188 tablet_bootstrap.cc:492] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:30.685423 16188 ts_tablet_manager.cc:1367] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.001s
I20240503 07:59:30.685964 16188 raft_consensus.cc:348] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:30.686195 16188 raft_consensus.cc:374] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.686264 16188 raft_consensus.cc:729] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.686488 16188 consensus_queue.cc:260] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:30.687999 16188 ts_tablet_manager.cc:1398] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:30.694811 16181 tablet_bootstrap.cc:492] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:30.695191 9483 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:30.695463 16181 ts_tablet_manager.cc:1367] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
I20240503 07:59:30.700496 16181 raft_consensus.cc:348] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:30.705574 7260 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:30.705719 16181 raft_consensus.cc:374] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.705776 16181 raft_consensus.cc:729] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.706746 16181 consensus_queue.cc:260] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:30.708451 16181 ts_tablet_manager.cc:1398] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:30.724926 6745 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:30.732657 15941 tablet.cc:1722] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.733094 15549 raft_consensus.cc:482] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.733192 15549 raft_consensus.cc:504] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:30.733599 15549 leader_election.cc:288] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:30.734452 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa18637596d4a39aff9a4e4a261228d" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:30.734592 6886 raft_consensus.cc:2454] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:30.735320 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa18637596d4a39aff9a4e4a261228d" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:30.735450 9767 raft_consensus.cc:2454] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:30.736270 9307 leader_election.cc:302] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:30.738374 16150 tablet.cc:1722] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.738435 15829 raft_consensus.cc:2781] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.738508 15829 raft_consensus.cc:482] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.738560 15829 raft_consensus.cc:3037] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.739764 15829 raft_consensus.cc:504] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:30.740126 15829 leader_election.cc:288] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:30.743222 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa18637596d4a39aff9a4e4a261228d" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:30.743364 9767 raft_consensus.cc:3037] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.744441 9767 raft_consensus.cc:2454] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:30.746143 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=6b7cebfa760b461385cc83a4b43a4de6] alter complete (version 3)
I20240503 07:59:30.746215 15506 raft_consensus.cc:482] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.746299 15506 raft_consensus.cc:504] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:30.746752 15506 leader_election.cc:288] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:30.747388 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef6b8a139ca4be5bc496b6678b6c1a2" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:30.747534 7019 raft_consensus.cc:2454] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:30.748081 6824 leader_election.cc:302] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:30.748256 15506 raft_consensus.cc:2781] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.748318 15506 raft_consensus.cc:482] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.748353 15506 raft_consensus.cc:3037] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.749056 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa18637596d4a39aff9a4e4a261228d" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:30.749156 6886 raft_consensus.cc:3037] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.749418 15506 raft_consensus.cc:504] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:30.749728 15506 leader_election.cc:288] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:30.750203 6886 raft_consensus.cc:2454] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:30.753706 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef6b8a139ca4be5bc496b6678b6c1a2" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:30.753854 7020 raft_consensus.cc:3037] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.754892 7020 raft_consensus.cc:2454] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:30.757542 9307 leader_election.cc:302] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:30.758118 15549 raft_consensus.cc:2781] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.758208 15549 raft_consensus.cc:686] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:30.758373 15549 consensus_queue.cc:237] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:30.758709 6824 leader_election.cc:302] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:30.758949 15993 raft_consensus.cc:2781] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.758955 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef6b8a139ca4be5bc496b6678b6c1a2" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:30.759040 15993 raft_consensus.cc:686] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:30.759076 9765 raft_consensus.cc:3037] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.759176 15993 consensus_queue.cc:237] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:30.760329 9765 raft_consensus.cc:2454] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:30.760568 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef6b8a139ca4be5bc496b6678b6c1a2" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:30.760689 9766 raft_consensus.cc:2362] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1. Re-sending same reply.
I20240503 07:59:30.761126 6743 catalog_manager.cc:5484] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:30.762594 15758 tablet.cc:1722] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.769894 6745 catalog_manager.cc:5484] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:30.789223 9504 consensus_queue.cc:237] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:30.790701 7278 raft_consensus.cc:1261] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:30.791029 9374 raft_consensus.cc:1261] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:30.791163 16205 consensus_queue.cc:1035] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:30.791451 16206 consensus_queue.cc:1035] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:30.793015 15471 raft_consensus.cc:2932] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:30.793352 7278 raft_consensus.cc:2932] T 6e051a416bb446df91e90eba13206341 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:30.793535 9374 raft_consensus.cc:2932] T 6e051a416bb446df91e90eba13206341 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:30.796537 6745 catalog_manager.cc:5484] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:30.799503 9745 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e051a416bb446df91e90eba13206341 with delete_type TABLET_DATA_TOMBSTONED (TS a372e98a670f49e9adca46ab5c836c4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:30.800545 16232 tablet_replica.cc:330] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:30.801508 16232 raft_consensus.cc:2229] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.801735 16232 raft_consensus.cc:2258] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.803833 16232 ts_tablet_manager.cc:1875] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:30.806604 16232 ts_tablet_manager.cc:1888] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:30.806691 16232 log.cc:1196] T 6e051a416bb446df91e90eba13206341 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/6e051a416bb446df91e90eba13206341
I20240503 07:59:30.807256 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 6e051a416bb446df91e90eba13206341 (table rebalancer_extra_table_12 [id=4b3fe402f6344742a8841ffd09a9f966]) successfully deleted
I20240503 07:59:30.856770 6738 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:30.856947 6738 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:30.859814 6738 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 0b6e89e693fd40d28d818bc97c0e546a
I20240503 07:59:30.860648 9483 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b6e89e693fd40d28d818bc97c0e546a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:30.860968 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b6e89e693fd40d28d818bc97c0e546a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:30.861497 16271 tablet_replica.cc:330] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:30.861701 16271 raft_consensus.cc:2229] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.861891 16271 raft_consensus.cc:2258] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.862530 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b6e89e693fd40d28d818bc97c0e546a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:30.862708 16271 ts_tablet_manager.cc:1875] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.862934 16272 tablet_replica.cc:330] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:30.863148 16272 raft_consensus.cc:2229] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:30.863431 16272 raft_consensus.cc:2258] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.863647 16270 tablet_replica.cc:330] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:30.863813 16270 raft_consensus.cc:2229] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.863955 16270 raft_consensus.cc:2258] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.864143 16272 ts_tablet_manager.cc:1875] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.866091 16270 ts_tablet_manager.cc:1875] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.866669 16272 ts_tablet_manager.cc:1888] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.866784 16272 log.cc:1196] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0b6e89e693fd40d28d818bc97c0e546a
I20240503 07:59:30.867138 16272 ts_tablet_manager.cc:1909] T 0b6e89e693fd40d28d818bc97c0e546a P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:30.868660 16271 ts_tablet_manager.cc:1888] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.868791 16271 log.cc:1196] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0b6e89e693fd40d28d818bc97c0e546a
I20240503 07:59:30.869135 16270 ts_tablet_manager.cc:1888] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.869256 16270 log.cc:1196] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/0b6e89e693fd40d28d818bc97c0e546a
I20240503 07:59:30.869298 16271 ts_tablet_manager.cc:1909] T 0b6e89e693fd40d28d818bc97c0e546a P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:30.869547 16270 ts_tablet_manager.cc:1909] T 0b6e89e693fd40d28d818bc97c0e546a P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:30.870446 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 0b6e89e693fd40d28d818bc97c0e546a (table rebalancer_dynamic_table_new_name [id=6b7cebfa760b461385cc83a4b43a4de6]) successfully deleted
I20240503 07:59:30.871114 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 0b6e89e693fd40d28d818bc97c0e546a (table rebalancer_dynamic_table_new_name [id=6b7cebfa760b461385cc83a4b43a4de6]) successfully deleted
I20240503 07:59:30.875401 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 0b6e89e693fd40d28d818bc97c0e546a (table rebalancer_dynamic_table_new_name [id=6b7cebfa760b461385cc83a4b43a4de6]) successfully deleted
I20240503 07:59:30.961468 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:30.967947 7260 tablet_service.cc:1467] Processing CreateTablet for tablet ab73e4b084de4cb394901cb55c0cf93a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=11de5bbc54da41f182f00ef568baa01b]), partition=
I20240503 07:59:30.967947 7131 tablet_service.cc:1467] Processing CreateTablet for tablet ab73e4b084de4cb394901cb55c0cf93a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=11de5bbc54da41f182f00ef568baa01b]), partition=
I20240503 07:59:30.968240 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab73e4b084de4cb394901cb55c0cf93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.968240 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab73e4b084de4cb394901cb55c0cf93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.970515 16185 tablet_bootstrap.cc:492] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:30.970515 16183 tablet_bootstrap.cc:492] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:30.970734 9616 tablet_service.cc:1467] Processing CreateTablet for tablet ab73e4b084de4cb394901cb55c0cf93a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=11de5bbc54da41f182f00ef568baa01b]), partition=
I20240503 07:59:30.970942 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab73e4b084de4cb394901cb55c0cf93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.971464 16185 tablet_bootstrap.cc:654] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.971462 16183 tablet_bootstrap.cc:654] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.973085 16275 tablet_bootstrap.cc:492] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:30.973898 16183 tablet_bootstrap.cc:492] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:30.973990 16183 ts_tablet_manager.cc:1367] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:30.974073 16185 tablet_bootstrap.cc:492] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:30.974099 16275 tablet_bootstrap.cc:654] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.974165 16185 ts_tablet_manager.cc:1367] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:30.974467 16183 raft_consensus.cc:348] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:30.974576 16183 raft_consensus.cc:374] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.974601 16183 raft_consensus.cc:729] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.974692 16185 raft_consensus.cc:348] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:30.974730 16183 consensus_queue.cc:260] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:30.974790 16185 raft_consensus.cc:374] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.974812 16185 raft_consensus.cc:729] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.974911 16185 consensus_queue.cc:260] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:30.975015 16183 ts_tablet_manager.cc:1398] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.975183 16185 ts_tablet_manager.cc:1398] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.977226 16275 tablet_bootstrap.cc:492] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:30.977331 16275 ts_tablet_manager.cc:1367] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:30.977737 16275 raft_consensus.cc:348] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:30.977850 16275 raft_consensus.cc:374] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.977880 16275 raft_consensus.cc:729] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.978298 16275 consensus_queue.cc:260] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:30.979423 16275 ts_tablet_manager.cc:1398] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:30.995559 14889 raft_consensus.cc:482] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.995726 14889 raft_consensus.cc:504] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:30.996208 14889 leader_election.cc:288] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:30.996687 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6d879affda4232af790bf7d38d5558" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:30.996857 7278 raft_consensus.cc:2454] T cb6d879affda4232af790bf7d38d5558 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:30.997174 7087 leader_election.cc:302] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:30.997310 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6d879affda4232af790bf7d38d5558" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:30.997395 14889 raft_consensus.cc:2781] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.997439 14889 raft_consensus.cc:482] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.997459 9505 raft_consensus.cc:2454] T cb6d879affda4232af790bf7d38d5558 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:30.997491 14889 raft_consensus.cc:3037] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.998827 14889 raft_consensus.cc:504] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:30.999214 14889 leader_election.cc:288] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:30.999750 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6d879affda4232af790bf7d38d5558" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:30.999881 7282 raft_consensus.cc:3037] T cb6d879affda4232af790bf7d38d5558 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.001088 7282 raft_consensus.cc:2454] T cb6d879affda4232af790bf7d38d5558 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:31.001528 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6d879affda4232af790bf7d38d5558" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:31.001673 9505 raft_consensus.cc:3037] T cb6d879affda4232af790bf7d38d5558 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.002622 9505 raft_consensus.cc:2454] T cb6d879affda4232af790bf7d38d5558 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:31.002650 7087 leader_election.cc:302] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:31.003139 14889 raft_consensus.cc:2781] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.003253 14889 raft_consensus.cc:686] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:31.003443 14889 consensus_queue.cc:237] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:31.006206 6739 catalog_manager.cc:5484] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:31.037498 9350 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:31.049428 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:31.049880 7131 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:31.054327 9616 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:31.054524 6998 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:31.058030 6867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:31.063203 9483 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:31.078271 7260 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:31.135963 6886 consensus_queue.cc:237] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:31.137439 9636 raft_consensus.cc:1261] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:31.137567 7282 raft_consensus.cc:1261] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:31.137930 15993 consensus_queue.cc:1035] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:31.138463 15993 consensus_queue.cc:1035] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:31.139911 15993 raft_consensus.cc:2932] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:31.140708 7282 raft_consensus.cc:2932] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:31.142086 9636 raft_consensus.cc:2932] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:31.146617 6739 catalog_manager.cc:5484] T 517c0e477a8c40fbb84c4cd3bb6ecabd P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:31.149637 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet 517c0e477a8c40fbb84c4cd3bb6ecabd with delete_type TABLET_DATA_TOMBSTONED (TS 3befbb2d206f4be89d3e07f38837374b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:31.149873 16271 tablet_replica.cc:330] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:31.150014 16271 raft_consensus.cc:2229] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:31.150147 16271 raft_consensus.cc:2258] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:31.150466 16207 consensus_queue.cc:1035] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.150691 16271 ts_tablet_manager.cc:1875] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:31.153113 16271 ts_tablet_manager.cc:1888] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:31.153187 16271 log.cc:1196] T 517c0e477a8c40fbb84c4cd3bb6ecabd P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/517c0e477a8c40fbb84c4cd3bb6ecabd
I20240503 07:59:31.155465 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 517c0e477a8c40fbb84c4cd3bb6ecabd (table rebalancer_extra_table_8 [id=914b54d652844204bb800dbd8a5be98a]) successfully deleted
I20240503 07:59:31.156742 16207 consensus_queue.cc:1035] T 0aa18637596d4a39aff9a4e4a261228d P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.159538 16132 raft_consensus.cc:482] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.159655 16132 raft_consensus.cc:504] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:31.160212 16132 leader_election.cc:288] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:31.160591 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab73e4b084de4cb394901cb55c0cf93a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:31.160740 7282 raft_consensus.cc:2454] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:31.161022 7087 leader_election.cc:302] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:31.160988 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab73e4b084de4cb394901cb55c0cf93a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:31.161139 9636 raft_consensus.cc:2454] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:31.161187 16132 raft_consensus.cc:2781] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.161234 16132 raft_consensus.cc:482] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.161266 16132 raft_consensus.cc:3037] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.162338 16132 raft_consensus.cc:504] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:31.162730 16132 leader_election.cc:288] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:31.163033 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab73e4b084de4cb394901cb55c0cf93a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:31.163062 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab73e4b084de4cb394901cb55c0cf93a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:31.163165 7282 raft_consensus.cc:3037] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.163192 9636 raft_consensus.cc:3037] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.164057 7282 raft_consensus.cc:2454] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:31.164220 9636 raft_consensus.cc:2454] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:31.164381 7087 leader_election.cc:302] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:31.164572 16132 raft_consensus.cc:2781] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.164646 16132 raft_consensus.cc:686] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:31.164805 16132 consensus_queue.cc:237] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:31.167896 6747 catalog_manager.cc:5484] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:31.199035 15993 consensus_queue.cc:1035] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.214531 15673 consensus_queue.cc:1035] T 7ef6b8a139ca4be5bc496b6678b6c1a2 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.281323 6744 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:31.281481 9616 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:31.282577 9350 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:31.290825 9635 raft_consensus.cc:1261] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:31.291394 16132 consensus_queue.cc:1035] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.291543 7282 raft_consensus.cc:1261] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:31.291914 16132 consensus_queue.cc:1035] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.292734 6998 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:31.295935 6744 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet ab73e4b084de4cb394901cb55c0cf93a (table rebalancer_dynamic_table [id=11de5bbc54da41f182f00ef568baa01b]). Expected version 1 got 0
I20240503 07:59:31.297933 7131 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:31.300426 16372 tablet.cc:1722] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.300525 6739 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet ab73e4b084de4cb394901cb55c0cf93a (table rebalancer_dynamic_table [id=11de5bbc54da41f182f00ef568baa01b]). Expected version 1 got 0
I20240503 07:59:31.300783 16372 tablet.cc:1620] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:31.303391 6744 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet ab73e4b084de4cb394901cb55c0cf93a (table rebalancer_dynamic_table [id=11de5bbc54da41f182f00ef568baa01b]). Expected version 1 got 0
I20240503 07:59:31.310988 6744 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=11de5bbc54da41f182f00ef568baa01b] alter complete (version 1)
I20240503 07:59:31.318517 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:31.319725 6739 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet ab73e4b084de4cb394901cb55c0cf93a (table rebalancer_dynamic_table [id=11de5bbc54da41f182f00ef568baa01b]). Expected version 1 got 0
I20240503 07:59:31.320590 9483 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:31.329759 7260 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:31.331249 6867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:31.326102 16373 tablet.cc:1722] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.338701 16373 tablet.cc:1620] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:31.344730 16374 tablet.cc:1722] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.345235 16374 tablet.cc:1620] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:31.402854 9636 consensus_queue.cc:237] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:31.404161 6888 raft_consensus.cc:1261] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:31.404611 15767 consensus_queue.cc:1035] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:31.404960 9374 raft_consensus.cc:1261] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:31.406368 15767 consensus_queue.cc:1035] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:31.407862 15485 raft_consensus.cc:2932] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:31.408360 6888 raft_consensus.cc:2932] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:31.409024 9374 raft_consensus.cc:2932] T 27139acdbdb54c248f35fcf9ec023f4d P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:31.411602 6739 catalog_manager.cc:5484] T 27139acdbdb54c248f35fcf9ec023f4d P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:31.416781 9483 tablet_service.cc:1514] Processing DeleteTablet for tablet 27139acdbdb54c248f35fcf9ec023f4d with delete_type TABLET_DATA_TOMBSTONED (TS cb73994515af4880b05383e78368bb4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:31.418270 16398 tablet_replica.cc:330] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:31.418630 16398 raft_consensus.cc:2229] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:31.419178 16398 raft_consensus.cc:2258] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:31.419986 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:31.420217 16398 ts_tablet_manager.cc:1875] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:31.425752 16398 ts_tablet_manager.cc:1888] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:31.425972 16398 log.cc:1196] T 27139acdbdb54c248f35fcf9ec023f4d P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/27139acdbdb54c248f35fcf9ec023f4d
I20240503 07:59:31.427502 16372 tablet.cc:1722] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:31.427881 16372 tablet.cc:1620] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:31.428705 16373 tablet.cc:1722] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:31.429008 16373 tablet.cc:1620] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:31.433632 16374 tablet.cc:1722] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:31.434015 16374 tablet.cc:1620] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:31.439242 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=11de5bbc54da41f182f00ef568baa01b] alter complete (version 2)
I20240503 07:59:31.447064 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 27139acdbdb54c248f35fcf9ec023f4d (table rebalancer_extra_table_18 [id=fb966e884afa4325a2dc1798eb47fbf4]) successfully deleted
I20240503 07:59:31.485941 16132 consensus_queue.cc:1035] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.491348 14889 consensus_queue.cc:1035] T cb6d879affda4232af790bf7d38d5558 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.543171 6745 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:31.543879 9616 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:31.550951 16372 tablet.cc:1722] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:31.551874 16373 tablet.cc:1722] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:31.552276 16374 tablet.cc:1722] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:31.559754 7131 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:31.560330 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:31.560330 7260 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:31.567817 9350 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:31.569615 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=11de5bbc54da41f182f00ef568baa01b] alter complete (version 3)
I20240503 07:59:31.573593 6998 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:31.580297 6745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:31.590540 6867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:31.598994 6998 tablet_service.cc:1467] Processing CreateTablet for tablet e700fe0935754d818e465345909a9c2c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65f04671509b48c89af8956f139749e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.599263 9351 tablet_service.cc:1467] Processing CreateTablet for tablet bed0aa049b8b4e7c926d6417acf4f068 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65f04671509b48c89af8956f139749e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.599381 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e700fe0935754d818e465345909a9c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.599588 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed0aa049b8b4e7c926d6417acf4f068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.599587 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 47ae0cf462874f26985099f1c5132ae0 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65f04671509b48c89af8956f139749e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.599843 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ae0cf462874f26985099f1c5132ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.602284 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 47ae0cf462874f26985099f1c5132ae0 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65f04671509b48c89af8956f139749e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.602289 9484 tablet_service.cc:1467] Processing CreateTablet for tablet bed0aa049b8b4e7c926d6417acf4f068 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65f04671509b48c89af8956f139749e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.602591 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed0aa049b8b4e7c926d6417acf4f068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.602591 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ae0cf462874f26985099f1c5132ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.605203 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 47ae0cf462874f26985099f1c5132ae0 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65f04671509b48c89af8956f139749e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.605249 9745 tablet_service.cc:1467] Processing CreateTablet for tablet e700fe0935754d818e465345909a9c2c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65f04671509b48c89af8956f139749e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.605502 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ae0cf462874f26985099f1c5132ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.605564 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e700fe0935754d818e465345909a9c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.608120 16455 tablet_bootstrap.cc:492] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:31.608399 9483 tablet_service.cc:1467] Processing CreateTablet for tablet e700fe0935754d818e465345909a9c2c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65f04671509b48c89af8956f139749e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.608677 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e700fe0935754d818e465345909a9c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.610059 16455 tablet_bootstrap.cc:654] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.610726 6867 tablet_service.cc:1467] Processing CreateTablet for tablet bed0aa049b8b4e7c926d6417acf4f068 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65f04671509b48c89af8956f139749e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.611063 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed0aa049b8b4e7c926d6417acf4f068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.611330 16455 tablet_bootstrap.cc:492] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:31.611418 16455 ts_tablet_manager.cc:1367] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:31.611907 16455 raft_consensus.cc:348] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.612023 16455 raft_consensus.cc:374] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.612053 16455 raft_consensus.cc:729] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.612181 16455 consensus_queue.cc:260] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.612493 16455 ts_tablet_manager.cc:1398] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.614835 16453 tablet_bootstrap.cc:492] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:31.615363 16454 tablet_bootstrap.cc:492] T 47ae0cf462874f26985099f1c5132ae0 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:31.616400 16454 tablet_bootstrap.cc:654] T 47ae0cf462874f26985099f1c5132ae0 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.616595 16457 tablet_bootstrap.cc:492] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:31.617352 16453 tablet_bootstrap.cc:654] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.618249 16320 raft_consensus.cc:482] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.618379 16320 raft_consensus.cc:504] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.618777 16320 leader_election.cc:288] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:31.619400 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e700fe0935754d818e465345909a9c2c" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
W20240503 07:59:31.620887 9703 leader_election.cc:341] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:31.623880 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e700fe0935754d818e465345909a9c2c" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
W20240503 07:59:31.624435 9702 leader_election.cc:341] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:31.624521 9702 leader_election.cc:302] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a; no voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a
I20240503 07:59:31.624747 16320 raft_consensus.cc:2726] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:31.626188 9483 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:31.627511 16457 tablet_bootstrap.cc:654] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.643533 16452 tablet_bootstrap.cc:492] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:31.650826 16453 tablet_bootstrap.cc:492] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:31.650949 16453 ts_tablet_manager.cc:1367] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240503 07:59:31.651446 16453 raft_consensus.cc:348] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:31.651568 16453 raft_consensus.cc:374] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.651600 16453 raft_consensus.cc:729] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.651717 16453 consensus_queue.cc:260] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:31.652050 16453 ts_tablet_manager.cc:1398] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.652240 16453 tablet_bootstrap.cc:492] T 47ae0cf462874f26985099f1c5132ae0 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:31.653805 16453 tablet_bootstrap.cc:654] T 47ae0cf462874f26985099f1c5132ae0 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.656044 16454 tablet_bootstrap.cc:492] T 47ae0cf462874f26985099f1c5132ae0 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:31.656148 16454 ts_tablet_manager.cc:1367] T 47ae0cf462874f26985099f1c5132ae0 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.000s
I20240503 07:59:31.656631 16454 raft_consensus.cc:348] T 47ae0cf462874f26985099f1c5132ae0 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:31.656751 16454 raft_consensus.cc:374] T 47ae0cf462874f26985099f1c5132ae0 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.656777 16454 raft_consensus.cc:729] T 47ae0cf462874f26985099f1c5132ae0 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.656900 16454 consensus_queue.cc:260] T 47ae0cf462874f26985099f1c5132ae0 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:31.657163 16454 ts_tablet_manager.cc:1398] T 47ae0cf462874f26985099f1c5132ae0 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.658330 16454 tablet_bootstrap.cc:492] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:31.660037 16457 tablet_bootstrap.cc:492] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:31.660147 16457 ts_tablet_manager.cc:1367] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20240503 07:59:31.659458 16454 tablet_bootstrap.cc:654] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.660627 16457 raft_consensus.cc:348] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:31.660750 16457 raft_consensus.cc:374] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.660775 16457 raft_consensus.cc:729] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.660903 16457 consensus_queue.cc:260] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:31.661201 16457 ts_tablet_manager.cc:1398] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:31.661770 16453 tablet_bootstrap.cc:492] T 47ae0cf462874f26985099f1c5132ae0 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:31.662060 16453 ts_tablet_manager.cc:1367] T 47ae0cf462874f26985099f1c5132ae0 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:31.662570 16453 raft_consensus.cc:348] T 47ae0cf462874f26985099f1c5132ae0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:31.662683 16453 raft_consensus.cc:374] T 47ae0cf462874f26985099f1c5132ae0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.662710 16453 raft_consensus.cc:729] T 47ae0cf462874f26985099f1c5132ae0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.662838 16453 consensus_queue.cc:260] T 47ae0cf462874f26985099f1c5132ae0 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:31.663285 16452 tablet_bootstrap.cc:654] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.664186 16453 ts_tablet_manager.cc:1398] T 47ae0cf462874f26985099f1c5132ae0 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:31.665388 16452 tablet_bootstrap.cc:492] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:31.665488 16452 ts_tablet_manager.cc:1367] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 07:59:31.667452 16452 raft_consensus.cc:348] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.668356 16452 raft_consensus.cc:374] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.668421 16452 raft_consensus.cc:729] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.669770 16452 consensus_queue.cc:260] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.670825 16452 ts_tablet_manager.cc:1398] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:31.671047 16452 tablet_bootstrap.cc:492] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:31.671247 15829 raft_consensus.cc:482] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.671336 15829 raft_consensus.cc:504] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:31.671586 6744 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:31.671703 6744 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:31.671824 15829 leader_election.cc:288] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:31.672412 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed0aa049b8b4e7c926d6417acf4f068" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:31.672550 6888 raft_consensus.cc:2454] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:31.675168 9307 leader_election.cc:302] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:31.675393 6744 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet ab73e4b084de4cb394901cb55c0cf93a
I20240503 07:59:31.675560 16207 raft_consensus.cc:2781] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.675618 16207 raft_consensus.cc:482] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.675653 16207 raft_consensus.cc:3037] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.676069 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet ab73e4b084de4cb394901cb55c0cf93a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:31 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:31.676542 9616 tablet_service.cc:1514] Processing DeleteTablet for tablet ab73e4b084de4cb394901cb55c0cf93a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:31 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:31.676889 16466 tablet_replica.cc:330] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:31.676975 16207 raft_consensus.cc:504] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:31.677191 16466 raft_consensus.cc:2229] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:31.677332 16466 raft_consensus.cc:2258] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:31.677356 16207 leader_election.cc:288] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:31.677510 7131 tablet_service.cc:1514] Processing DeleteTablet for tablet ab73e4b084de4cb394901cb55c0cf93a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:31 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:31.678277 16466 ts_tablet_manager.cc:1875] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:31.679248 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed0aa049b8b4e7c926d6417acf4f068" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:31.679559 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed0aa049b8b4e7c926d6417acf4f068" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
W20240503 07:59:31.682840 9310 leader_election.cc:341] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:31.682996 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed0aa049b8b4e7c926d6417acf4f068" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
W20240503 07:59:31.683095 9310 leader_election.cc:341] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:31.683122 6888 raft_consensus.cc:3037] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.684192 6888 raft_consensus.cc:2454] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:31.685601 9307 leader_election.cc:302] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, c4218904dd7e42d2be95b901a6e5db92; no voters: cb73994515af4880b05383e78368bb4a
I20240503 07:59:31.685809 16207 raft_consensus.cc:2781] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.685916 16207 raft_consensus.cc:686] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:31.686061 16207 consensus_queue.cc:237] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:31.686399 16466 ts_tablet_manager.cc:1888] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:31.686539 16466 log.cc:1196] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/ab73e4b084de4cb394901cb55c0cf93a
I20240503 07:59:31.686632 16454 tablet_bootstrap.cc:492] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:31.686750 16454 ts_tablet_manager.cc:1367] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.002s
I20240503 07:59:31.687284 16454 raft_consensus.cc:348] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:31.687398 16454 raft_consensus.cc:374] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.687424 16454 raft_consensus.cc:729] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.687546 16454 consensus_queue.cc:260] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:31.687700 16466 ts_tablet_manager.cc:1909] T ab73e4b084de4cb394901cb55c0cf93a P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:31.687880 16454 ts_tablet_manager.cc:1398] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:31.688053 16454 tablet_bootstrap.cc:492] T e700fe0935754d818e465345909a9c2c P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:31.688503 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet ab73e4b084de4cb394901cb55c0cf93a (table rebalancer_dynamic_table_new_name [id=11de5bbc54da41f182f00ef568baa01b]) successfully deleted
I20240503 07:59:31.689066 6744 catalog_manager.cc:5484] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:31.692232 16454 tablet_bootstrap.cc:654] T e700fe0935754d818e465345909a9c2c P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.692689 16452 tablet_bootstrap.cc:654] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.693470 16454 tablet_bootstrap.cc:492] T e700fe0935754d818e465345909a9c2c P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:31.693565 16454 ts_tablet_manager.cc:1367] T e700fe0935754d818e465345909a9c2c P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 07:59:31.693833 16452 tablet_bootstrap.cc:492] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:31.693928 16452 ts_tablet_manager.cc:1367] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240503 07:59:31.694002 16454 raft_consensus.cc:348] T e700fe0935754d818e465345909a9c2c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.694135 16454 raft_consensus.cc:374] T e700fe0935754d818e465345909a9c2c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.694188 16454 raft_consensus.cc:729] T e700fe0935754d818e465345909a9c2c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.694334 16454 consensus_queue.cc:260] T e700fe0935754d818e465345909a9c2c P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.694768 16454 ts_tablet_manager.cc:1398] T e700fe0935754d818e465345909a9c2c P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:31.694450 16452 raft_consensus.cc:348] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:31.695106 16452 raft_consensus.cc:374] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.695186 16452 raft_consensus.cc:729] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.695326 16452 consensus_queue.cc:260] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:31.695631 16452 ts_tablet_manager.cc:1398] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:31.695765 16467 tablet_replica.cc:330] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:31.695916 16467 raft_consensus.cc:2229] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:31.696115 16467 raft_consensus.cc:2258] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:31.696380 16465 tablet_replica.cc:330] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:31.696549 16465 raft_consensus.cc:2229] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:31.696702 16465 raft_consensus.cc:2258] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:31.696784 16467 ts_tablet_manager.cc:1875] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:31.697531 16465 ts_tablet_manager.cc:1875] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:31.700052 16467 ts_tablet_manager.cc:1888] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:31.700198 16467 log.cc:1196] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ab73e4b084de4cb394901cb55c0cf93a
I20240503 07:59:31.700609 16467 ts_tablet_manager.cc:1909] T ab73e4b084de4cb394901cb55c0cf93a P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:31.701366 16465 ts_tablet_manager.cc:1888] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:31.701457 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet ab73e4b084de4cb394901cb55c0cf93a (table rebalancer_dynamic_table_new_name [id=11de5bbc54da41f182f00ef568baa01b]) successfully deleted
I20240503 07:59:31.701547 16465 log.cc:1196] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ab73e4b084de4cb394901cb55c0cf93a
I20240503 07:59:31.701881 16465 ts_tablet_manager.cc:1909] T ab73e4b084de4cb394901cb55c0cf93a P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:31.702558 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet ab73e4b084de4cb394901cb55c0cf93a (table rebalancer_dynamic_table_new_name [id=11de5bbc54da41f182f00ef568baa01b]) successfully deleted
I20240503 07:59:31.723125 9636 consensus_queue.cc:237] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:31.724625 7020 raft_consensus.cc:1261] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:31.724627 6888 raft_consensus.cc:1261] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:31.725071 15767 consensus_queue.cc:1035] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:31.725260 15485 consensus_queue.cc:1035] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:31.727767 15767 raft_consensus.cc:2932] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:31.729928 7020 raft_consensus.cc:2932] T 23759e1c50b9402980461599687904b6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:31.730711 6888 raft_consensus.cc:2932] T 23759e1c50b9402980461599687904b6 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:31.731349 6745 catalog_manager.cc:5484] T 23759e1c50b9402980461599687904b6 P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:31.736033 9483 tablet_service.cc:1514] Processing DeleteTablet for tablet 23759e1c50b9402980461599687904b6 with delete_type TABLET_DATA_TOMBSTONED (TS cb73994515af4880b05383e78368bb4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:31.736307 16398 tablet_replica.cc:330] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:31.736481 16398 raft_consensus.cc:2229] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:31.736609 16398 raft_consensus.cc:2258] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:31.737135 16398 ts_tablet_manager.cc:1875] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:31.739382 16398 ts_tablet_manager.cc:1888] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:31.739449 16398 log.cc:1196] T 23759e1c50b9402980461599687904b6 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/23759e1c50b9402980461599687904b6
I20240503 07:59:31.740051 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 23759e1c50b9402980461599687904b6 (table rebalancer_extra_table_0 [id=62034a44b5d9412f853d552a90206f04]) successfully deleted
I20240503 07:59:31.745755 16405 raft_consensus.cc:482] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.745867 16405 raft_consensus.cc:504] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:31.746330 16405 leader_election.cc:288] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:31.746743 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ae0cf462874f26985099f1c5132ae0" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:31.746819 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ae0cf462874f26985099f1c5132ae0" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:31.746927 9374 raft_consensus.cc:2454] T 47ae0cf462874f26985099f1c5132ae0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:31.746938 9505 raft_consensus.cc:2454] T 47ae0cf462874f26985099f1c5132ae0 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:31.747220 6956 leader_election.cc:302] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:31.747422 16405 raft_consensus.cc:2781] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.747478 16405 raft_consensus.cc:482] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.747516 16405 raft_consensus.cc:3037] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.748687 16405 raft_consensus.cc:504] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:31.749527 16405 leader_election.cc:288] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:31.749826 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ae0cf462874f26985099f1c5132ae0" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:31.749956 9505 raft_consensus.cc:3037] T 47ae0cf462874f26985099f1c5132ae0 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.750964 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ae0cf462874f26985099f1c5132ae0" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:31.751029 9505 raft_consensus.cc:2454] T 47ae0cf462874f26985099f1c5132ae0 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:31.751103 9374 raft_consensus.cc:3037] T 47ae0cf462874f26985099f1c5132ae0 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.751498 6956 leader_election.cc:302] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:31.751830 16405 raft_consensus.cc:2781] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.751914 16405 raft_consensus.cc:686] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:31.752041 16405 consensus_queue.cc:237] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:31.752233 9374 raft_consensus.cc:2454] T 47ae0cf462874f26985099f1c5132ae0 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:31.754549 6746 catalog_manager.cc:5484] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:31.777482 6746 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:31.784047 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc2019f93dc4566a94475a526343278 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0a318cb238554004a25c2daa3811d4ca]), partition=
I20240503 07:59:31.784408 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc2019f93dc4566a94475a526343278. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.786825 16517 tablet_bootstrap.cc:492] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:31.786893 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc2019f93dc4566a94475a526343278 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0a318cb238554004a25c2daa3811d4ca]), partition=
I20240503 07:59:31.787155 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc2019f93dc4566a94475a526343278. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.787945 16517 tablet_bootstrap.cc:654] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.789479 16517 tablet_bootstrap.cc:492] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:31.789562 16517 ts_tablet_manager.cc:1367] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:31.789649 16519 tablet_bootstrap.cc:492] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:31.789870 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc2019f93dc4566a94475a526343278 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0a318cb238554004a25c2daa3811d4ca]), partition=
I20240503 07:59:31.790056 16517 raft_consensus.cc:348] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.790125 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc2019f93dc4566a94475a526343278. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.790150 16517 raft_consensus.cc:374] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.790172 16517 raft_consensus.cc:729] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.790292 16517 consensus_queue.cc:260] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.790751 16517 ts_tablet_manager.cc:1398] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.791680 16519 tablet_bootstrap.cc:654] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.792986 16455 tablet_bootstrap.cc:492] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:31.793922 16455 tablet_bootstrap.cc:654] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.796448 16455 tablet_bootstrap.cc:492] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:31.796566 16455 ts_tablet_manager.cc:1367] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:31.797133 16519 tablet_bootstrap.cc:492] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:31.797103 16455 raft_consensus.cc:348] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.797241 16519 ts_tablet_manager.cc:1367] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:31.797250 16455 raft_consensus.cc:374] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.797283 16455 raft_consensus.cc:729] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.797412 16455 consensus_queue.cc:260] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.797746 16519 raft_consensus.cc:348] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.797865 16519 raft_consensus.cc:374] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.797899 16519 raft_consensus.cc:729] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.797933 16455 ts_tablet_manager.cc:1398] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.798019 16519 consensus_queue.cc:260] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.798312 16519 ts_tablet_manager.cc:1398] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.879036 16489 raft_consensus.cc:482] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.879163 16489 raft_consensus.cc:504] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.879571 16489 leader_election.cc:288] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:31.879940 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc2019f93dc4566a94475a526343278" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:31.880007 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc2019f93dc4566a94475a526343278" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:31.880102 7282 raft_consensus.cc:2454] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:31.880215 9765 raft_consensus.cc:2454] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:31.880842 9572 leader_election.cc:302] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:31.881026 16489 raft_consensus.cc:2781] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.881079 16489 raft_consensus.cc:482] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.881112 16489 raft_consensus.cc:3037] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.882203 16489 raft_consensus.cc:504] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.882594 16489 leader_election.cc:288] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:31.882997 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc2019f93dc4566a94475a526343278" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:31.883047 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc2019f93dc4566a94475a526343278" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:31.883116 15487 raft_consensus.cc:482] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.883150 7282 raft_consensus.cc:3037] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.883173 9765 raft_consensus.cc:3037] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.883210 15487 raft_consensus.cc:504] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.883646 15487 leader_election.cc:288] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:31.884172 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e700fe0935754d818e465345909a9c2c" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:31.884299 9766 raft_consensus.cc:2454] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:31.884306 7282 raft_consensus.cc:2454] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:31.884302 9765 raft_consensus.cc:2454] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:31.884588 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e700fe0935754d818e465345909a9c2c" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:31.884718 9572 leader_election.cc:302] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:31.884735 9505 raft_consensus.cc:2454] T e700fe0935754d818e465345909a9c2c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:31.885004 16489 raft_consensus.cc:2781] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.885104 16489 raft_consensus.cc:686] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:31.885090 6956 leader_election.cc:302] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:31.885280 16489 consensus_queue.cc:237] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.885365 15487 raft_consensus.cc:2781] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.885424 15487 raft_consensus.cc:482] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.885458 15487 raft_consensus.cc:3037] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.886619 15487 raft_consensus.cc:504] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.886920 15487 leader_election.cc:288] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:31.887312 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e700fe0935754d818e465345909a9c2c" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:31.887434 9505 raft_consensus.cc:3037] T e700fe0935754d818e465345909a9c2c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.888407 9505 raft_consensus.cc:2454] T e700fe0935754d818e465345909a9c2c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:31.888749 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e700fe0935754d818e465345909a9c2c" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:31.888767 6956 leader_election.cc:302] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:31.888897 9765 raft_consensus.cc:3037] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.888954 15487 raft_consensus.cc:2781] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.888955 6739 catalog_manager.cc:5484] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:31.889094 15487 raft_consensus.cc:686] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:31.889230 15487 consensus_queue.cc:237] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:31.889928 9765 raft_consensus.cc:2454] T e700fe0935754d818e465345909a9c2c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:31.891511 6745 catalog_manager.cc:5484] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:31.915469 9616 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:31.915642 9350 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:31.915958 6999 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:31.917701 7260 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:31.919653 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:31.923331 7131 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:31.942656 6867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:31.942967 9483 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:31.950480 16206 raft_consensus.cc:482] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.950639 16206 raft_consensus.cc:504] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:31.951140 16206 leader_election.cc:288] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:31.951514 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed0aa049b8b4e7c926d6417acf4f068" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:31.952322 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed0aa049b8b4e7c926d6417acf4f068" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:31.952505 6886 raft_consensus.cc:2379] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb73994515af4880b05383e78368bb4a in current term 1: Already voted for candidate 3befbb2d206f4be89d3e07f38837374b in this term.
I20240503 07:59:31.953200 9438 leader_election.cc:302] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb73994515af4880b05383e78368bb4a; no voters: 3befbb2d206f4be89d3e07f38837374b, c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:31.953431 16206 raft_consensus.cc:3037] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.954838 16206 raft_consensus.cc:2726] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:31.999748 6745 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:32.005863 7282 raft_consensus.cc:1261] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:32.006297 15767 consensus_queue.cc:1035] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.007562 9765 raft_consensus.cc:1261] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:32.010373 16489 consensus_queue.cc:1035] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.012140 6745 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet 8fc2019f93dc4566a94475a526343278 (table rebalancer_dynamic_table [id=0a318cb238554004a25c2daa3811d4ca]). Expected version 1 got 0
I20240503 07:59:32.014143 16373 tablet.cc:1722] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:32.014268 6745 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 8fc2019f93dc4566a94475a526343278 (table rebalancer_dynamic_table [id=0a318cb238554004a25c2daa3811d4ca]). Expected version 1 got 0
I20240503 07:59:32.014541 16373 tablet.cc:1620] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:32.018678 6739 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 8fc2019f93dc4566a94475a526343278 (table rebalancer_dynamic_table [id=0a318cb238554004a25c2daa3811d4ca]). Expected version 1 got 0
I20240503 07:59:32.022171 16374 tablet.cc:1722] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:32.022547 16374 tablet.cc:1620] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:32.022891 16368 mvcc.cc:205] Tried to move back new op lower bound from 7023506112529002496 to 7023506112047443968. Current Snapshot: MvccSnapshot[applied={T|T < 7023506112529002496}]
I20240503 07:59:32.024327 16573 tablet.cc:1722] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:32.024662 16573 tablet.cc:1620] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:32.026602 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=0a318cb238554004a25c2daa3811d4ca] alter complete (version 1)
I20240503 07:59:32.031930 7282 consensus_queue.cc:237] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:32.033290 7150 raft_consensus.cc:1261] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:32.033290 9765 raft_consensus.cc:1261] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:32.033694 14839 consensus_queue.cc:1035] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:32.033879 14440 consensus_queue.cc:1035] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:32.035899 16582 raft_consensus.cc:2932] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:32.036330 9765 raft_consensus.cc:2932] T ba1de17d15b74c4497e649aadc8569d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:32.036319 7150 raft_consensus.cc:2932] T ba1de17d15b74c4497e649aadc8569d2 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:32.039840 6745 catalog_manager.cc:5484] T ba1de17d15b74c4497e649aadc8569d2 P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:32.043665 6999 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1de17d15b74c4497e649aadc8569d2 with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:32.044052 16587 tablet_replica.cc:330] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:32.044216 16587 raft_consensus.cc:2229] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.044402 16587 raft_consensus.cc:2258] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.045012 16587 ts_tablet_manager.cc:1875] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:32.047607 16587 ts_tablet_manager.cc:1888] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:32.047777 16587 log.cc:1196] T ba1de17d15b74c4497e649aadc8569d2 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba1de17d15b74c4497e649aadc8569d2
I20240503 07:59:32.048373 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet ba1de17d15b74c4497e649aadc8569d2 (table rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]) successfully deleted
I20240503 07:59:32.077267 9357 consensus_queue.cc:237] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
W20240503 07:59:32.079018 9309 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b -> Peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): Couldn't send request to peer 9c085a3241f14cb7ba1332b508f4b57d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ef6defd02954e009565bae98f5e8418. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:32.079100 7150 raft_consensus.cc:1261] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:32.079100 9765 raft_consensus.cc:1261] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:32.079481 15829 consensus_queue.cc:1035] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:32.079696 16207 consensus_queue.cc:1035] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:32.082077 15829 raft_consensus.cc:2932] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:32.082532 9765 raft_consensus.cc:2932] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:32.086005 6739 catalog_manager.cc:5484] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:32.088459 7150 raft_consensus.cc:2932] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:32.105363 16207 consensus_queue.cc:1035] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.108881 7282 consensus_queue.cc:237] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:32.110641 16610 consensus_queue.cc:1035] T bed0aa049b8b4e7c926d6417acf4f068 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:32.110771 7215 consensus_peers.cc:485] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64d33ff4164b49a7948aa5afee40bf3f. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:32.111115 9505 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:32.111490 7150 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:32.111493 16582 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:32.112051 16582 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:32.114988 14440 raft_consensus.cc:2932] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:32.116784 9505 raft_consensus.cc:2932] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:32.116858 7150 raft_consensus.cc:2932] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:32.117928 6742 catalog_manager.cc:5484] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index 5 to 6, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:32.131521 6745 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:32.138191 16373 tablet.cc:1722] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.138618 16373 tablet.cc:1620] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:32.138725 16374 tablet.cc:1722] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.139036 16374 tablet.cc:1620] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:32.139225 16573 tablet.cc:1722] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.139535 16573 tablet.cc:1620] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:32.146822 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=0a318cb238554004a25c2daa3811d4ca] alter complete (version 2)
I20240503 07:59:32.148348 9505 consensus_queue.cc:237] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.149907 9765 raft_consensus.cc:1261] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:32.150328 16653 consensus_queue.cc:1035] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:32.150508 6888 raft_consensus.cc:1261] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:32.150890 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5388c1fa25bc4f19a2aef9d8960e88aa. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:32.151033 16653 consensus_queue.cc:1035] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:32.152818 16461 raft_consensus.cc:2932] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:32.153265 6888 raft_consensus.cc:2932] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:32.156388 6742 catalog_manager.cc:5484] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index -1 to 2, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:32.156522 9765 raft_consensus.cc:2932] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:32.179657 9636 consensus_queue.cc:237] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:32.181303 9505 raft_consensus.cc:1261] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:32.181326 9572 consensus_peers.cc:485] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 -> Peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): Couldn't send request to peer 00a3f8e159c64327832d79fb32cdcca8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 730eb09f729844ce9f5952c46031c0d8. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:32.181447 9765 raft_consensus.cc:1261] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:32.181777 16489 consensus_queue.cc:1035] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:32.182126 15767 consensus_queue.cc:1035] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:32.184590 16489 raft_consensus.cc:2932] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:32.186785 9765 raft_consensus.cc:2932] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:32.188364 9505 raft_consensus.cc:2932] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:32.188814 6739 catalog_manager.cc:5484] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:32.251305 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:32.259373 16373 tablet.cc:1722] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.265269 16374 tablet.cc:1722] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.266507 16573 tablet.cc:1722] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.268443 6747 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=0a318cb238554004a25c2daa3811d4ca] alter complete (version 3)
I20240503 07:59:32.372885 15487 consensus_queue.cc:1035] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.374528 15487 consensus_queue.cc:1035] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.382983 6747 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:32.383177 6747 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:32.385923 6747 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 8fc2019f93dc4566a94475a526343278
I20240503 07:59:32.386437 9616 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc2019f93dc4566a94475a526343278 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:32.386631 9745 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc2019f93dc4566a94475a526343278 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:32.387032 16771 tablet_replica.cc:330] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:32.388206 16771 raft_consensus.cc:2229] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.388406 16771 raft_consensus.cc:2258] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.389665 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc2019f93dc4566a94475a526343278 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:32.389991 16774 tablet_replica.cc:330] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:32.390235 16774 raft_consensus.cc:2229] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.390743 16774 raft_consensus.cc:2258] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.391825 16771 ts_tablet_manager.cc:1875] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.394444 16774 ts_tablet_manager.cc:1875] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.396262 7131 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:32.398141 16771 ts_tablet_manager.cc:1888] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.398293 16771 log.cc:1196] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/8fc2019f93dc4566a94475a526343278
I20240503 07:59:32.398689 16771 ts_tablet_manager.cc:1909] T 8fc2019f93dc4566a94475a526343278 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:32.399595 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 8fc2019f93dc4566a94475a526343278 (table rebalancer_dynamic_table_new_name [id=0a318cb238554004a25c2daa3811d4ca]) successfully deleted
I20240503 07:59:32.404752 9350 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:32.407019 16774 ts_tablet_manager.cc:1888] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.407605 16774 log.cc:1196] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8fc2019f93dc4566a94475a526343278
I20240503 07:59:32.408063 16774 ts_tablet_manager.cc:1909] T 8fc2019f93dc4566a94475a526343278 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:32.409086 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 8fc2019f93dc4566a94475a526343278 (table rebalancer_dynamic_table_new_name [id=0a318cb238554004a25c2daa3811d4ca]) successfully deleted
I20240503 07:59:32.409512 16405 consensus_queue.cc:1035] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.410490 16770 tablet_replica.cc:330] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:32.410750 16770 raft_consensus.cc:2229] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:32.411039 16770 raft_consensus.cc:2258] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.412911 16770 ts_tablet_manager.cc:1875] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.419693 16770 ts_tablet_manager.cc:1888] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.419806 16770 log.cc:1196] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/8fc2019f93dc4566a94475a526343278
I20240503 07:59:32.420552 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:32.422008 16770 ts_tablet_manager.cc:1909] T 8fc2019f93dc4566a94475a526343278 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:32.423959 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:32.428604 9616 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:32.432121 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 8fc2019f93dc4566a94475a526343278 (table rebalancer_dynamic_table_new_name [id=0a318cb238554004a25c2daa3811d4ca]) successfully deleted
I20240503 07:59:32.434620 15487 consensus_queue.cc:1035] T e700fe0935754d818e465345909a9c2c P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.441690 6867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:32.442406 9351 tablet_service.cc:1467] Processing CreateTablet for tablet 1c23c1c8b02147b780c9e1aebc48f24a (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e16004005302445ba90feb044ec20393]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.442528 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 56badbfc1fab4848ac79ebe2f20781b6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e16004005302445ba90feb044ec20393]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.442700 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 16a4b20f1cc24ed09fdaaa2f9764e709 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e16004005302445ba90feb044ec20393]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.442857 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56badbfc1fab4848ac79ebe2f20781b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.443074 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 16a4b20f1cc24ed09fdaaa2f9764e709 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e16004005302445ba90feb044ec20393]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.443332 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a4b20f1cc24ed09fdaaa2f9764e709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.444779 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 56badbfc1fab4848ac79ebe2f20781b6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e16004005302445ba90feb044ec20393]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.444923 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 1c23c1c8b02147b780c9e1aebc48f24a (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e16004005302445ba90feb044ec20393]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.445075 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56badbfc1fab4848ac79ebe2f20781b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.445132 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c23c1c8b02147b780c9e1aebc48f24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.446100 16777 tablet_bootstrap.cc:492] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:32.447124 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 16a4b20f1cc24ed09fdaaa2f9764e709 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e16004005302445ba90feb044ec20393]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.447407 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a4b20f1cc24ed09fdaaa2f9764e709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.447616 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 56badbfc1fab4848ac79ebe2f20781b6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e16004005302445ba90feb044ec20393]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.447849 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56badbfc1fab4848ac79ebe2f20781b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.442772 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c23c1c8b02147b780c9e1aebc48f24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.449751 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a4b20f1cc24ed09fdaaa2f9764e709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.453588 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 1c23c1c8b02147b780c9e1aebc48f24a (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e16004005302445ba90feb044ec20393]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.453897 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c23c1c8b02147b780c9e1aebc48f24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.456128 7260 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:32.456562 16780 tablet_bootstrap.cc:492] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:32.457187 16777 tablet_bootstrap.cc:654] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.459152 16780 tablet_bootstrap.cc:654] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.475690 16779 tablet_bootstrap.cc:492] T 56badbfc1fab4848ac79ebe2f20781b6 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:32.476620 9483 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:32.477175 16781 tablet_bootstrap.cc:492] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:32.478401 16781 tablet_bootstrap.cc:654] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.478694 16779 tablet_bootstrap.cc:654] T 56badbfc1fab4848ac79ebe2f20781b6 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.479955 16777 tablet_bootstrap.cc:492] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:32.480118 16777 ts_tablet_manager.cc:1367] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240503 07:59:32.480989 16777 raft_consensus.cc:348] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:32.481269 16777 raft_consensus.cc:374] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.481349 16777 raft_consensus.cc:729] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.481544 16777 consensus_queue.cc:260] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:32.484716 16777 ts_tablet_manager.cc:1398] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:32.484966 16777 tablet_bootstrap.cc:492] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:32.486296 16777 tablet_bootstrap.cc:654] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.489043 16780 tablet_bootstrap.cc:492] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:32.489042 16777 tablet_bootstrap.cc:492] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:32.489145 16780 ts_tablet_manager.cc:1367] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240503 07:59:32.489145 16777 ts_tablet_manager.cc:1367] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:32.489377 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:32.489663 16780 raft_consensus.cc:348] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.489763 16780 raft_consensus.cc:374] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.489652 16777 raft_consensus.cc:348] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.489791 16780 raft_consensus.cc:729] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.489805 16777 raft_consensus.cc:374] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.489838 16777 raft_consensus.cc:729] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.489931 16780 consensus_queue.cc:260] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.489966 16777 consensus_queue.cc:260] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.490252 16780 ts_tablet_manager.cc:1398] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.490407 16777 ts_tablet_manager.cc:1398] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.490757 16777 tablet_bootstrap.cc:492] T 56badbfc1fab4848ac79ebe2f20781b6 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:32.491842 16777 tablet_bootstrap.cc:654] T 56badbfc1fab4848ac79ebe2f20781b6 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.492656 16779 tablet_bootstrap.cc:492] T 56badbfc1fab4848ac79ebe2f20781b6 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:32.492822 16779 ts_tablet_manager.cc:1367] T 56badbfc1fab4848ac79ebe2f20781b6 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 07:59:32.494128 16779 raft_consensus.cc:348] T 56badbfc1fab4848ac79ebe2f20781b6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:32.496287 16779 raft_consensus.cc:374] T 56badbfc1fab4848ac79ebe2f20781b6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.496359 16779 raft_consensus.cc:729] T 56badbfc1fab4848ac79ebe2f20781b6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.496533 16779 consensus_queue.cc:260] T 56badbfc1fab4848ac79ebe2f20781b6 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:32.497613 16779 ts_tablet_manager.cc:1398] T 56badbfc1fab4848ac79ebe2f20781b6 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:32.497642 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2db8226e774540d19122551ac896fa4c]), partition=
I20240503 07:59:32.497865 16779 tablet_bootstrap.cc:492] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:32.497921 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.499122 16779 tablet_bootstrap.cc:654] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.500599 16779 tablet_bootstrap.cc:492] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:32.500684 16779 ts_tablet_manager.cc:1367] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.501168 16779 raft_consensus.cc:348] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:32.501293 16779 raft_consensus.cc:374] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.501327 16779 raft_consensus.cc:729] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.501545 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2db8226e774540d19122551ac896fa4c]), partition=
I20240503 07:59:32.501739 16777 tablet_bootstrap.cc:492] T 56badbfc1fab4848ac79ebe2f20781b6 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:32.501806 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.501835 16777 ts_tablet_manager.cc:1367] T 56badbfc1fab4848ac79ebe2f20781b6 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:32.502265 16777 raft_consensus.cc:348] T 56badbfc1fab4848ac79ebe2f20781b6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:32.502622 16779 consensus_queue.cc:260] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:32.502846 16777 raft_consensus.cc:374] T 56badbfc1fab4848ac79ebe2f20781b6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.502902 16777 raft_consensus.cc:729] T 56badbfc1fab4848ac79ebe2f20781b6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.503557 16779 ts_tablet_manager.cc:1398] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.503612 16781 tablet_bootstrap.cc:492] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:32.503743 16781 ts_tablet_manager.cc:1367] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.002s
I20240503 07:59:32.503758 16777 consensus_queue.cc:260] T 56badbfc1fab4848ac79ebe2f20781b6 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:32.504432 16777 ts_tablet_manager.cc:1398] T 56badbfc1fab4848ac79ebe2f20781b6 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:32.504412 16781 raft_consensus.cc:348] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:32.504556 16781 raft_consensus.cc:374] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.504606 16781 raft_consensus.cc:729] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.504777 16781 consensus_queue.cc:260] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:32.505188 16781 ts_tablet_manager.cc:1398] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:32.505944 16781 tablet_bootstrap.cc:492] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:32.506570 16790 tablet_bootstrap.cc:492] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:32.506561 9351 tablet_service.cc:1467] Processing CreateTablet for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2db8226e774540d19122551ac896fa4c]), partition=
I20240503 07:59:32.506820 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.507047 16781 tablet_bootstrap.cc:654] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.507952 16790 tablet_bootstrap.cc:654] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.513442 16781 tablet_bootstrap.cc:492] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:32.513566 16781 ts_tablet_manager.cc:1367] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:32.515461 16790 tablet_bootstrap.cc:492] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:32.515650 16790 ts_tablet_manager.cc:1367] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:32.516263 16790 raft_consensus.cc:348] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.516383 16781 raft_consensus.cc:348] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.516503 16790 raft_consensus.cc:374] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.516557 16781 raft_consensus.cc:374] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.516575 16790 raft_consensus.cc:729] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.516655 16781 raft_consensus.cc:729] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.516742 16790 consensus_queue.cc:260] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.517467 16781 consensus_queue.cc:260] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.517891 16781 ts_tablet_manager.cc:1398] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:32.519202 16790 ts_tablet_manager.cc:1398] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:32.518146 16781 tablet_bootstrap.cc:492] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:32.520946 16781 tablet_bootstrap.cc:654] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.526902 16781 tablet_bootstrap.cc:492] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:32.527011 16781 ts_tablet_manager.cc:1367] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 07:59:32.527523 16781 raft_consensus.cc:348] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.527650 16781 raft_consensus.cc:374] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.527686 16781 raft_consensus.cc:729] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.527809 16781 consensus_queue.cc:260] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.528077 16781 ts_tablet_manager.cc:1398] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:32.529814 6999 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:32.530795 16778 tablet_bootstrap.cc:492] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:32.531971 16778 tablet_bootstrap.cc:654] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.534878 16778 tablet_bootstrap.cc:492] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:32.534943 16789 tablet_bootstrap.cc:492] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:32.535022 16778 ts_tablet_manager.cc:1367] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:32.535522 16778 raft_consensus.cc:348] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:32.535679 16778 raft_consensus.cc:374] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.535733 16778 raft_consensus.cc:729] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.535883 16778 consensus_queue.cc:260] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:32.536190 16778 ts_tablet_manager.cc:1398] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.544818 16789 tablet_bootstrap.cc:654] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.559103 16789 tablet_bootstrap.cc:492] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:32.559238 16789 ts_tablet_manager.cc:1367] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 07:59:32.561313 16789 raft_consensus.cc:348] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.561542 16789 raft_consensus.cc:374] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.561623 16789 raft_consensus.cc:729] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.561828 16789 consensus_queue.cc:260] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.562247 16789 ts_tablet_manager.cc:1398] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.562528 16796 ts_tablet_manager.cc:902] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d: Initiating tablet copy from peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:32.563129 16796 tablet_copy_client.cc:308] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37815
I20240503 07:59:32.563686 9394 tablet_copy_service.cc:140] P 3befbb2d206f4be89d3e07f38837374b: Received BeginTabletCopySession request for tablet 4ef6defd02954e009565bae98f5e8418 from peer 9c085a3241f14cb7ba1332b508f4b57d ({username='slave'} at 127.0.99.130:52175)
I20240503 07:59:32.563808 9394 tablet_copy_service.cc:161] P 3befbb2d206f4be89d3e07f38837374b: Beginning new tablet copy session on tablet 4ef6defd02954e009565bae98f5e8418 from peer 9c085a3241f14cb7ba1332b508f4b57d at {username='slave'} at 127.0.99.130:52175: session id = 9c085a3241f14cb7ba1332b508f4b57d-4ef6defd02954e009565bae98f5e8418
I20240503 07:59:32.564728 9394 tablet_copy_source_session.cc:205] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:32.566236 16796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef6defd02954e009565bae98f5e8418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.568881 16206 raft_consensus.cc:482] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.568987 16206 raft_consensus.cc:504] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:32.569224 16796 tablet_copy_client.cc:783] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:32.569383 16796 tablet_copy_client.cc:647] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:32.569455 16206 leader_election.cc:288] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:32.570256 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56badbfc1fab4848ac79ebe2f20781b6" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:32.570776 7149 raft_consensus.cc:2454] T 56badbfc1fab4848ac79ebe2f20781b6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:32.571066 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56badbfc1fab4848ac79ebe2f20781b6" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:32.571209 7282 raft_consensus.cc:2454] T 56badbfc1fab4848ac79ebe2f20781b6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:32.571730 9441 leader_election.cc:302] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:32.572113 16206 raft_consensus.cc:2781] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.572176 16206 raft_consensus.cc:482] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.572224 16206 raft_consensus.cc:3037] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.573065 16206 raft_consensus.cc:504] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:32.573426 16206 leader_election.cc:288] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:32.573555 16796 tablet_copy_client.cc:523] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:32.573786 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56badbfc1fab4848ac79ebe2f20781b6" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:32.573931 7149 raft_consensus.cc:3037] T 56badbfc1fab4848ac79ebe2f20781b6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.574816 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56badbfc1fab4848ac79ebe2f20781b6" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:32.574877 16796 tablet_bootstrap.cc:492] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:32.574942 7282 raft_consensus.cc:3037] T 56badbfc1fab4848ac79ebe2f20781b6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.575054 7149 raft_consensus.cc:2454] T 56badbfc1fab4848ac79ebe2f20781b6 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:32.576017 7282 raft_consensus.cc:2454] T 56badbfc1fab4848ac79ebe2f20781b6 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:32.576570 9441 leader_election.cc:302] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:32.576764 16206 raft_consensus.cc:2781] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.576858 16206 raft_consensus.cc:686] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:32.577000 16206 consensus_queue.cc:237] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:32.578730 16796 tablet_bootstrap.cc:492] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:32.578953 16796 tablet_bootstrap.cc:492] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap complete.
I20240503 07:59:32.579079 16796 ts_tablet_manager.cc:1367] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240503 07:59:32.579465 16796 raft_consensus.cc:348] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:32.579560 16796 raft_consensus.cc:729] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.579509 6739 catalog_manager.cc:5484] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:32.579672 16796 consensus_queue.cc:260] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:32.579931 16796 ts_tablet_manager.cc:1398] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:32.581223 9394 tablet_copy_service.cc:342] P 3befbb2d206f4be89d3e07f38837374b: Request end of tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-4ef6defd02954e009565bae98f5e8418 received from {username='slave'} at 127.0.99.130:52175
I20240503 07:59:32.581306 9394 tablet_copy_service.cc:433] P 3befbb2d206f4be89d3e07f38837374b: ending tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-4ef6defd02954e009565bae98f5e8418 on tablet 4ef6defd02954e009565bae98f5e8418 with peer 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:32.599788 16207 raft_consensus.cc:482] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.599928 16207 raft_consensus.cc:504] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.600466 16207 leader_election.cc:288] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:32.600908 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ffd6e83ec64340b7d6bd2012eeaaa9" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:32.601064 9635 raft_consensus.cc:2454] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:32.601339 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ffd6e83ec64340b7d6bd2012eeaaa9" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:32.601420 9307 leader_election.cc:302] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:32.601480 6888 raft_consensus.cc:2454] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:32.601519 7282 consensus_queue.cc:237] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.601614 16207 raft_consensus.cc:2781] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.601665 16207 raft_consensus.cc:482] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.601699 16207 raft_consensus.cc:3037] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.602866 6888 raft_consensus.cc:1261] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:32.603055 16207 raft_consensus.cc:504] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.603441 16207 leader_election.cc:288] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:32.603714 9374 raft_consensus.cc:1261] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:32.603813 16582 consensus_queue.cc:1035] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:32.604418 14839 consensus_queue.cc:1035] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:32.605041 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ffd6e83ec64340b7d6bd2012eeaaa9" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:32.605172 6886 raft_consensus.cc:3037] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.606159 6886 raft_consensus.cc:2454] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:32.606537 9307 leader_election.cc:302] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:32.606736 16610 raft_consensus.cc:2781] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.606863 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ffd6e83ec64340b7d6bd2012eeaaa9" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:32.606935 16610 raft_consensus.cc:686] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:32.606994 9635 raft_consensus.cc:3037] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.607069 16610 consensus_queue.cc:237] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.608065 9635 raft_consensus.cc:2454] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:32.610282 6739 catalog_manager.cc:5484] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:32.613549 14839 raft_consensus.cc:2932] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:32.614550 6888 raft_consensus.cc:2932] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:32.616184 9374 raft_consensus.cc:2932] T 6e63db30c0454722b6a794a9259a67c0 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:32.618953 6747 catalog_manager.cc:5484] T 6e63db30c0454722b6a794a9259a67c0 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:32.624567 16823 ts_tablet_manager.cc:902] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:32.625164 16823 tablet_copy_client.cc:308] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:32.627148 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet 64d33ff4164b49a7948aa5afee40bf3f from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:43259)
I20240503 07:59:32.627280 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet 64d33ff4164b49a7948aa5afee40bf3f from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:43259: session id = 159177b3caeb434baa98e04136d050b2-64d33ff4164b49a7948aa5afee40bf3f
I20240503 07:59:32.627827 16821 raft_consensus.cc:482] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.627928 16821 raft_consensus.cc:504] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.628327 7302 tablet_copy_source_session.cc:205] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:32.628806 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c23c1c8b02147b780c9e1aebc48f24a" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:32.628968 9766 raft_consensus.cc:2454] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:32.629256 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e63db30c0454722b6a794a9259a67c0 with delete_type TABLET_DATA_TOMBSTONED (TS 00a3f8e159c64327832d79fb32cdcca8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:32.629691 16833 tablet_replica.cc:330] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:32.629724 9310 leader_election.cc:302] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:32.629920 16207 raft_consensus.cc:2781] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.629984 16207 raft_consensus.cc:482] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.630033 16207 raft_consensus.cc:3037] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.630105 16833 raft_consensus.cc:2229] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.630335 16833 raft_consensus.cc:2258] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.631000 16833 ts_tablet_manager.cc:1875] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:32.631282 16207 raft_consensus.cc:504] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.631707 16207 leader_election.cc:288] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:32.632081 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d33ff4164b49a7948aa5afee40bf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.632390 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c23c1c8b02147b780c9e1aebc48f24a" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:32.632508 7149 raft_consensus.cc:3037] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.632963 16821 leader_election.cc:288] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:32.633493 7149 raft_consensus.cc:2454] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:32.633708 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c23c1c8b02147b780c9e1aebc48f24a" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:32.633854 7150 raft_consensus.cc:2362] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1. Re-sending same reply.
I20240503 07:59:32.633914 9310 leader_election.cc:302] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:32.634074 16833 ts_tablet_manager.cc:1888] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:32.634145 16833 log.cc:1196] T 6e63db30c0454722b6a794a9259a67c0 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/6e63db30c0454722b6a794a9259a67c0
I20240503 07:59:32.634318 16207 raft_consensus.cc:2781] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.634321 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c23c1c8b02147b780c9e1aebc48f24a" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:32.634472 16207 raft_consensus.cc:686] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:32.634498 9766 raft_consensus.cc:3037] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.634619 16207 consensus_queue.cc:237] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.634982 16823 tablet_copy_client.cc:783] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:32.635131 16823 tablet_copy_client.cc:647] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:32.635557 9766 raft_consensus.cc:2454] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:32.636013 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 6e63db30c0454722b6a794a9259a67c0 (table rebalance_test_table_11 [id=9caa58273688477ab18820b0590675ab]) successfully deleted
I20240503 07:59:32.637107 6747 catalog_manager.cc:5484] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:32.637825 16823 tablet_copy_client.cc:523] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:32.639452 16823 tablet_bootstrap.cc:492] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:32.661912 16823 tablet_bootstrap.cc:492] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:32.662268 16823 tablet_bootstrap.cc:492] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:32.662479 16823 ts_tablet_manager.cc:1367] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.023s user 0.023s sys 0.000s
I20240503 07:59:32.662890 16823 raft_consensus.cc:348] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:32.663002 16823 raft_consensus.cc:729] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.663141 16823 consensus_queue.cc:260] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:32.664649 16823 ts_tablet_manager.cc:1398] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:32.666440 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-64d33ff4164b49a7948aa5afee40bf3f received from {username='slave'} at 127.0.99.135:43259
I20240503 07:59:32.666535 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session 159177b3caeb434baa98e04136d050b2-64d33ff4164b49a7948aa5afee40bf3f on tablet 64d33ff4164b49a7948aa5afee40bf3f with peer 159177b3caeb434baa98e04136d050b2
I20240503 07:59:32.719312 16872 ts_tablet_manager.cc:902] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:32.720064 16610 raft_consensus.cc:482] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.720185 16872 tablet_copy_client.cc:308] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:45435
I20240503 07:59:32.720193 16610 raft_consensus.cc:504] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:32.720710 16610 leader_election.cc:288] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:32.720779 9525 tablet_copy_service.cc:140] P cb73994515af4880b05383e78368bb4a: Received BeginTabletCopySession request for tablet 5388c1fa25bc4f19a2aef9d8960e88aa from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:48135)
I20240503 07:59:32.720893 9525 tablet_copy_service.cc:161] P cb73994515af4880b05383e78368bb4a: Beginning new tablet copy session on tablet 5388c1fa25bc4f19a2aef9d8960e88aa from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:48135: session id = 3befbb2d206f4be89d3e07f38837374b-5388c1fa25bc4f19a2aef9d8960e88aa
I20240503 07:59:32.721693 9525 tablet_copy_source_session.cc:205] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:32.722651 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a4b20f1cc24ed09fdaaa2f9764e709" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:32.722817 7150 raft_consensus.cc:2454] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:32.722849 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a4b20f1cc24ed09fdaaa2f9764e709" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:32.722980 7282 raft_consensus.cc:2454] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:32.726557 9310 leader_election.cc:302] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 3befbb2d206f4be89d3e07f38837374b; no voters:
I20240503 07:59:32.727042 16610 raft_consensus.cc:2781] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.727119 16610 raft_consensus.cc:482] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.727159 16610 raft_consensus.cc:3037] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.728482 16610 raft_consensus.cc:504] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:32.728897 16610 leader_election.cc:288] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:32.732473 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a4b20f1cc24ed09fdaaa2f9764e709" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:32.732630 7282 raft_consensus.cc:3037] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.733989 7282 raft_consensus.cc:2454] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:32.734248 16872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5388c1fa25bc4f19a2aef9d8960e88aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.734668 9310 leader_election.cc:302] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:32.734680 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a4b20f1cc24ed09fdaaa2f9764e709" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:32.734840 7150 raft_consensus.cc:3037] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.736047 7150 raft_consensus.cc:2454] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:32.738087 6744 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:32.738457 16610 raft_consensus.cc:2781] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.738559 16610 raft_consensus.cc:686] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:32.738694 16610 consensus_queue.cc:237] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:32.744550 6745 catalog_manager.cc:5484] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:32.745606 9635 raft_consensus.cc:1261] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:32.745926 6888 raft_consensus.cc:1261] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:32.745824 16872 tablet_copy_client.cc:783] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:32.746206 16872 tablet_copy_client.cc:647] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:32.746714 16610 consensus_queue.cc:1035] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.747040 16821 consensus_queue.cc:1035] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.751694 16872 tablet_copy_client.cc:523] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:32.751758 6741 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 (table rebalancer_dynamic_table [id=2db8226e774540d19122551ac896fa4c]). Expected version 1 got 0
I20240503 07:59:32.753468 6741 catalog_manager.cc:5576] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655) does not have the latest schema for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 (table rebalancer_dynamic_table [id=2db8226e774540d19122551ac896fa4c]). Expected version 1 got 0
I20240503 07:59:32.753557 16872 tablet_bootstrap.cc:492] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:32.757434 16881 tablet.cc:1722] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:32.757848 16881 tablet.cc:1620] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:32.758436 16373 tablet.cc:1722] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:32.758749 16373 tablet.cc:1620] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:32.760242 6745 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 (table rebalancer_dynamic_table [id=2db8226e774540d19122551ac896fa4c]). Expected version 1 got 0
I20240503 07:59:32.768162 16884 tablet.cc:1722] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:32.769079 16884 tablet.cc:1620] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:32.770162 6744 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=2db8226e774540d19122551ac896fa4c] alter complete (version 1)
I20240503 07:59:32.770419 6745 catalog_manager.cc:5576] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655) does not have the latest schema for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 (table rebalancer_dynamic_table [id=2db8226e774540d19122551ac896fa4c]). Expected version 1 got 0
I20240503 07:59:32.770459 16872 tablet_bootstrap.cc:492] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:32.770509 16883 ts_tablet_manager.cc:902] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8: Initiating tablet copy from peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:32.770717 16872 tablet_bootstrap.cc:492] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:32.770889 16872 ts_tablet_manager.cc:1367] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240503 07:59:32.771093 16883 tablet_copy_client.cc:308] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:43369
I20240503 07:59:32.771510 16872 raft_consensus.cc:348] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.771641 16872 raft_consensus.cc:729] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.771723 9656 tablet_copy_service.cc:140] P 159177b3caeb434baa98e04136d050b2: Received BeginTabletCopySession request for tablet 730eb09f729844ce9f5952c46031c0d8 from peer 00a3f8e159c64327832d79fb32cdcca8 ({username='slave'} at 127.0.99.131:53011)
I20240503 07:59:32.771826 9656 tablet_copy_service.cc:161] P 159177b3caeb434baa98e04136d050b2: Beginning new tablet copy session on tablet 730eb09f729844ce9f5952c46031c0d8 from peer 00a3f8e159c64327832d79fb32cdcca8 at {username='slave'} at 127.0.99.131:53011: session id = 00a3f8e159c64327832d79fb32cdcca8-730eb09f729844ce9f5952c46031c0d8
I20240503 07:59:32.772881 9656 tablet_copy_source_session.cc:205] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:32.771775 16872 consensus_queue.cc:260] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:32.773372 16872 ts_tablet_manager.cc:1398] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:32.775199 9525 tablet_copy_service.cc:342] P cb73994515af4880b05383e78368bb4a: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-5388c1fa25bc4f19a2aef9d8960e88aa received from {username='slave'} at 127.0.99.133:48135
I20240503 07:59:32.775300 9525 tablet_copy_service.cc:433] P cb73994515af4880b05383e78368bb4a: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-5388c1fa25bc4f19a2aef9d8960e88aa on tablet 5388c1fa25bc4f19a2aef9d8960e88aa with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:32.775329 16883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730eb09f729844ce9f5952c46031c0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.778728 16883 tablet_copy_client.cc:783] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:32.779402 16883 tablet_copy_client.cc:647] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:32.782707 16883 tablet_copy_client.cc:523] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:32.787959 16883 tablet_bootstrap.cc:492] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:32.793043 16883 tablet_bootstrap.cc:492] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:32.793413 16883 tablet_bootstrap.cc:492] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap complete.
I20240503 07:59:32.793650 16883 ts_tablet_manager.cc:1367] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 07:59:32.794230 16883 raft_consensus.cc:348] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:32.794489 16883 raft_consensus.cc:729] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.794698 16883 consensus_queue.cc:260] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:32.795197 16883 ts_tablet_manager.cc:1398] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:32.795703 9656 tablet_copy_service.cc:342] P 159177b3caeb434baa98e04136d050b2: Request end of tablet copy session 00a3f8e159c64327832d79fb32cdcca8-730eb09f729844ce9f5952c46031c0d8 received from {username='slave'} at 127.0.99.131:53011
I20240503 07:59:32.795799 9656 tablet_copy_service.cc:433] P 159177b3caeb434baa98e04136d050b2: ending tablet copy session 00a3f8e159c64327832d79fb32cdcca8-730eb09f729844ce9f5952c46031c0d8 on tablet 730eb09f729844ce9f5952c46031c0d8 with peer 00a3f8e159c64327832d79fb32cdcca8
I20240503 07:59:32.813205 7129 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:32.818038 9616 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:32.822845 7260 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:32.826370 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:32.828573 6999 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:32.829306 6867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:32.843335 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:32.860565 9483 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:32.875926 6744 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:32.885820 16881 tablet.cc:1722] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.886205 16881 tablet.cc:1620] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:32.887517 16884 tablet.cc:1722] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.887861 16884 tablet.cc:1620] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:32.892091 16373 tablet.cc:1722] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.892448 16373 tablet.cc:1620] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:32.894814 6744 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=2db8226e774540d19122551ac896fa4c] alter complete (version 2)
I20240503 07:59:32.980085 16206 consensus_queue.cc:1035] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.984647 16461 consensus_queue.cc:1035] T 56badbfc1fab4848ac79ebe2f20781b6 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.996661 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:33.005653 16881 tablet.cc:1722] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:33.006335 16884 tablet.cc:1722] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:33.007174 16373 tablet.cc:1722] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:33.013024 6742 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=2db8226e774540d19122551ac896fa4c] alter complete (version 3)
I20240503 07:59:33.067050 9616 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:33.071596 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:33.074524 7260 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:33.077595 7129 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:33.085392 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:33.091405 6867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:33.092568 6999 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:33.109910 7020 raft_consensus.cc:1203] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:33.115622 9483 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:33.116638 6744 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:33.116802 6744 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:33.121444 6744 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 09ffd6e83ec64340b7d6bd2012eeaaa9
I20240503 07:59:33.122309 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:33 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:33.122782 9351 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:33 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:33.123080 9616 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:33 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:33.129231 16959 tablet_replica.cc:330] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:33.129516 16959 raft_consensus.cc:2229] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:33.129822 16959 raft_consensus.cc:2258] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.130507 16958 tablet_replica.cc:330] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:33.130726 16959 ts_tablet_manager.cc:1875] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:33.130740 16958 raft_consensus.cc:2229] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:33.130859 16958 raft_consensus.cc:2258] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.131139 16960 tablet_replica.cc:330] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:33.131562 16958 ts_tablet_manager.cc:1875] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:33.134604 16958 ts_tablet_manager.cc:1888] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:33.134282 16960 raft_consensus.cc:2229] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:33.135279 16959 ts_tablet_manager.cc:1888] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:33.135448 16959 log.cc:1196] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/09ffd6e83ec64340b7d6bd2012eeaaa9
I20240503 07:59:33.135794 16959 ts_tablet_manager.cc:1909] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:33.137168 16960 raft_consensus.cc:2258] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.137357 16958 log.cc:1196] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/09ffd6e83ec64340b7d6bd2012eeaaa9
I20240503 07:59:33.137701 16958 ts_tablet_manager.cc:1909] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:33.138643 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 (table rebalancer_dynamic_table_new_name [id=2db8226e774540d19122551ac896fa4c]) successfully deleted
I20240503 07:59:33.138904 16960 ts_tablet_manager.cc:1875] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:33.140012 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 (table rebalancer_dynamic_table_new_name [id=2db8226e774540d19122551ac896fa4c]) successfully deleted
I20240503 07:59:33.141712 16960 ts_tablet_manager.cc:1888] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:33.141803 16960 log.cc:1196] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/09ffd6e83ec64340b7d6bd2012eeaaa9
I20240503 07:59:33.142042 16960 ts_tablet_manager.cc:1909] T 09ffd6e83ec64340b7d6bd2012eeaaa9 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:33.142711 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 09ffd6e83ec64340b7d6bd2012eeaaa9 (table rebalancer_dynamic_table_new_name [id=2db8226e774540d19122551ac896fa4c]) successfully deleted
I20240503 07:59:33.154870 9635 raft_consensus.cc:1203] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 07:59:33.159806 16207 consensus_queue.cc:1035] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.165189 16207 consensus_queue.cc:1035] T 1c23c1c8b02147b780c9e1aebc48f24a P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.178995 9504 consensus_queue.cc:237] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:33.182588 9766 raft_consensus.cc:1261] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:33.182972 6888 raft_consensus.cc:1261] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:33.183091 16206 consensus_queue.cc:1035] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:33.183429 16461 consensus_queue.cc:1035] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:33.185643 16206 raft_consensus.cc:2932] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:33.187283 6886 raft_consensus.cc:2932] T a8c22545fc9f4b248c2fdc568967f47f P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:33.188724 6738 catalog_manager.cc:5484] T a8c22545fc9f4b248c2fdc568967f47f P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:33.190449 9766 raft_consensus.cc:2932] T a8c22545fc9f4b248c2fdc568967f47f P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:33.197199 9616 tablet_service.cc:1514] Processing DeleteTablet for tablet a8c22545fc9f4b248c2fdc568967f47f with delete_type TABLET_DATA_TOMBSTONED (TS 159177b3caeb434baa98e04136d050b2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:33.197491 16960 tablet_replica.cc:330] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:33.197664 16960 raft_consensus.cc:2229] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:33.197799 16960 raft_consensus.cc:2258] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.198371 16960 ts_tablet_manager.cc:1875] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:33.200978 16960 ts_tablet_manager.cc:1888] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:33.201066 16960 log.cc:1196] T a8c22545fc9f4b248c2fdc568967f47f P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a8c22545fc9f4b248c2fdc568967f47f
I20240503 07:59:33.202230 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet a8c22545fc9f4b248c2fdc568967f47f (table rebalancer_extra_table_16 [id=a0ee201ab5cc47b798438b4276757b44]) successfully deleted
I20240503 07:59:33.223838 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:33.230830 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 235d98cc3a5e4913991decba34badef8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8dd32be971c24de28551c2176484df1f]), partition=
I20240503 07:59:33.231146 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235d98cc3a5e4913991decba34badef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.233719 16994 tablet_bootstrap.cc:492] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:33.235061 16994 tablet_bootstrap.cc:654] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.236446 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 235d98cc3a5e4913991decba34badef8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8dd32be971c24de28551c2176484df1f]), partition=
I20240503 07:59:33.236583 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 235d98cc3a5e4913991decba34badef8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8dd32be971c24de28551c2176484df1f]), partition=
I20240503 07:59:33.236723 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235d98cc3a5e4913991decba34badef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.236828 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235d98cc3a5e4913991decba34badef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.240062 16997 tablet_bootstrap.cc:492] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:33.240394 16994 tablet_bootstrap.cc:492] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:33.240659 16994 ts_tablet_manager.cc:1367] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 07:59:33.241219 16997 tablet_bootstrap.cc:654] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.242444 16997 tablet_bootstrap.cc:492] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:33.242542 16997 ts_tablet_manager.cc:1367] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:33.243009 16997 raft_consensus.cc:348] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.243139 16997 raft_consensus.cc:374] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.243173 16997 raft_consensus.cc:729] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.243325 16997 consensus_queue.cc:260] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.243623 16997 ts_tablet_manager.cc:1398] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.243809 16994 raft_consensus.cc:348] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.243918 16994 raft_consensus.cc:374] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.243948 16994 raft_consensus.cc:729] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.244866 16994 consensus_queue.cc:260] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.246320 16994 ts_tablet_manager.cc:1398] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240503 07:59:33.246754 16996 tablet_bootstrap.cc:492] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:33.247859 16996 tablet_bootstrap.cc:654] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.249233 16996 tablet_bootstrap.cc:492] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:33.249331 16996 ts_tablet_manager.cc:1367] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:33.249826 16996 raft_consensus.cc:348] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.249958 16996 raft_consensus.cc:374] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.249990 16996 raft_consensus.cc:729] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.250115 16996 consensus_queue.cc:260] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.250432 16996 ts_tablet_manager.cc:1398] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.256541 16207 consensus_queue.cc:1035] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.260673 16821 consensus_queue.cc:1035] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.289337 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:33.297401 6867 tablet_service.cc:1467] Processing CreateTablet for tablet a327249bd2224ecd8754b69f228fe6d4 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=31ad4a6607f54abdb74612f11554a7ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.297784 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a327249bd2224ecd8754b69f228fe6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.298730 9374 raft_consensus.cc:1203] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:33.300083 9483 tablet_service.cc:1467] Processing CreateTablet for tablet a327249bd2224ecd8754b69f228fe6d4 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=31ad4a6607f54abdb74612f11554a7ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.300194 9745 tablet_service.cc:1467] Processing CreateTablet for tablet a327249bd2224ecd8754b69f228fe6d4 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=31ad4a6607f54abdb74612f11554a7ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.300364 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a327249bd2224ecd8754b69f228fe6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.300493 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a327249bd2224ecd8754b69f228fe6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.301584 9616 tablet_service.cc:1467] Processing CreateTablet for tablet ab08cdb57b5649fbad1ec5792c78b33e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=31ad4a6607f54abdb74612f11554a7ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.301872 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab08cdb57b5649fbad1ec5792c78b33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.303014 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 10c65e6d1cb74c90b2087b4111315e74 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=31ad4a6607f54abdb74612f11554a7ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.303244 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c65e6d1cb74c90b2087b4111315e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.305387 16994 tablet_bootstrap.cc:492] T a327249bd2224ecd8754b69f228fe6d4 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:33.306522 16994 tablet_bootstrap.cc:654] T a327249bd2224ecd8754b69f228fe6d4 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.307731 17033 tablet_bootstrap.cc:492] T a327249bd2224ecd8754b69f228fe6d4 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:33.307957 16994 tablet_bootstrap.cc:492] T a327249bd2224ecd8754b69f228fe6d4 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:33.308070 16994 ts_tablet_manager.cc:1367] T a327249bd2224ecd8754b69f228fe6d4 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:33.308679 17032 tablet_bootstrap.cc:492] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:33.308939 17033 tablet_bootstrap.cc:654] T a327249bd2224ecd8754b69f228fe6d4 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.309032 16994 raft_consensus.cc:348] T a327249bd2224ecd8754b69f228fe6d4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.309186 16994 raft_consensus.cc:374] T a327249bd2224ecd8754b69f228fe6d4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.309218 16994 raft_consensus.cc:729] T a327249bd2224ecd8754b69f228fe6d4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.309525 16994 consensus_queue.cc:260] T a327249bd2224ecd8754b69f228fe6d4 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.309748 17032 tablet_bootstrap.cc:654] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.309890 16994 ts_tablet_manager.cc:1398] T a327249bd2224ecd8754b69f228fe6d4 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:33.310263 17033 tablet_bootstrap.cc:492] T a327249bd2224ecd8754b69f228fe6d4 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:33.310379 17033 ts_tablet_manager.cc:1367] T a327249bd2224ecd8754b69f228fe6d4 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:33.310837 17033 raft_consensus.cc:348] T a327249bd2224ecd8754b69f228fe6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.310923 17033 raft_consensus.cc:374] T a327249bd2224ecd8754b69f228fe6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.310947 17033 raft_consensus.cc:729] T a327249bd2224ecd8754b69f228fe6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.311070 17033 consensus_queue.cc:260] T a327249bd2224ecd8754b69f228fe6d4 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.311347 17033 ts_tablet_manager.cc:1398] T a327249bd2224ecd8754b69f228fe6d4 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.312073 7129 tablet_service.cc:1467] Processing CreateTablet for tablet ab08cdb57b5649fbad1ec5792c78b33e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=31ad4a6607f54abdb74612f11554a7ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.312384 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab08cdb57b5649fbad1ec5792c78b33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.312412 7260 tablet_service.cc:1467] Processing CreateTablet for tablet ab08cdb57b5649fbad1ec5792c78b33e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=31ad4a6607f54abdb74612f11554a7ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.312494 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 10c65e6d1cb74c90b2087b4111315e74 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=31ad4a6607f54abdb74612f11554a7ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.312697 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab08cdb57b5649fbad1ec5792c78b33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.312707 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c65e6d1cb74c90b2087b4111315e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.313267 16132 raft_consensus.cc:482] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.313369 16132 raft_consensus.cc:504] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.313766 16132 leader_election.cc:288] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:33.314285 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235d98cc3a5e4913991decba34badef8" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:33.314440 9504 raft_consensus.cc:2454] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:33.315006 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 10c65e6d1cb74c90b2087b4111315e74 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=31ad4a6607f54abdb74612f11554a7ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.315304 16997 tablet_bootstrap.cc:492] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:33.315323 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c65e6d1cb74c90b2087b4111315e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.316318 16994 tablet_bootstrap.cc:492] T 10c65e6d1cb74c90b2087b4111315e74 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:33.316550 16997 tablet_bootstrap.cc:654] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.317463 16994 tablet_bootstrap.cc:654] T 10c65e6d1cb74c90b2087b4111315e74 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.317674 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235d98cc3a5e4913991decba34badef8" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:33.317718 7087 leader_election.cc:302] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:33.317802 7278 raft_consensus.cc:2454] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:33.318048 14889 raft_consensus.cc:2781] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.318101 14889 raft_consensus.cc:482] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.318138 14889 raft_consensus.cc:3037] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.319265 16996 tablet_bootstrap.cc:492] T ab08cdb57b5649fbad1ec5792c78b33e P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:33.319242 14889 raft_consensus.cc:504] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.319624 14889 leader_election.cc:288] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:33.320293 16996 tablet_bootstrap.cc:654] T ab08cdb57b5649fbad1ec5792c78b33e P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.321303 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235d98cc3a5e4913991decba34badef8" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:33.321430 9504 raft_consensus.cc:3037] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.323311 9504 raft_consensus.cc:2454] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:33.323417 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235d98cc3a5e4913991decba34badef8" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:33.323539 7278 raft_consensus.cc:3037] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.323695 7087 leader_election.cc:302] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:33.323859 16132 raft_consensus.cc:2781] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.323920 16132 raft_consensus.cc:686] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:33.324040 16132 consensus_queue.cc:237] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.324640 7278 raft_consensus.cc:2454] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:33.325891 16996 tablet_bootstrap.cc:492] T ab08cdb57b5649fbad1ec5792c78b33e P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:33.325989 16996 ts_tablet_manager.cc:1367] T ab08cdb57b5649fbad1ec5792c78b33e P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:33.326537 16996 raft_consensus.cc:348] T ab08cdb57b5649fbad1ec5792c78b33e P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.326664 16996 raft_consensus.cc:374] T ab08cdb57b5649fbad1ec5792c78b33e P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.326699 16996 raft_consensus.cc:729] T ab08cdb57b5649fbad1ec5792c78b33e P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.326802 16996 consensus_queue.cc:260] T ab08cdb57b5649fbad1ec5792c78b33e P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.327100 16996 ts_tablet_manager.cc:1398] T ab08cdb57b5649fbad1ec5792c78b33e P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.328819 7150 raft_consensus.cc:1203] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:33.328997 16997 tablet_bootstrap.cc:492] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:33.329097 16997 ts_tablet_manager.cc:1367] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:33.329654 16997 raft_consensus.cc:348] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.329773 16997 raft_consensus.cc:374] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.329806 16997 raft_consensus.cc:729] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.330796 17036 tablet_bootstrap.cc:492] T ab08cdb57b5649fbad1ec5792c78b33e P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:33.331300 16997 consensus_queue.cc:260] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.331998 17036 tablet_bootstrap.cc:654] T ab08cdb57b5649fbad1ec5792c78b33e P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.332795 16997 ts_tablet_manager.cc:1398] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:33.333108 16997 tablet_bootstrap.cc:492] T 10c65e6d1cb74c90b2087b4111315e74 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:33.333745 17036 tablet_bootstrap.cc:492] T ab08cdb57b5649fbad1ec5792c78b33e P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:33.333926 17036 ts_tablet_manager.cc:1367] T ab08cdb57b5649fbad1ec5792c78b33e P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:33.334334 16997 tablet_bootstrap.cc:654] T 10c65e6d1cb74c90b2087b4111315e74 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.334530 17036 raft_consensus.cc:348] T ab08cdb57b5649fbad1ec5792c78b33e P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.334728 17036 raft_consensus.cc:374] T ab08cdb57b5649fbad1ec5792c78b33e P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.334826 17036 raft_consensus.cc:729] T ab08cdb57b5649fbad1ec5792c78b33e P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.335011 17036 consensus_queue.cc:260] T ab08cdb57b5649fbad1ec5792c78b33e P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.335362 17036 ts_tablet_manager.cc:1398] T ab08cdb57b5649fbad1ec5792c78b33e P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.335670 17036 tablet_bootstrap.cc:492] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:33.336401 16997 tablet_bootstrap.cc:492] T 10c65e6d1cb74c90b2087b4111315e74 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:33.336586 16997 ts_tablet_manager.cc:1367] T 10c65e6d1cb74c90b2087b4111315e74 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:33.336943 17036 tablet_bootstrap.cc:654] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.338011 17032 tablet_bootstrap.cc:492] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:33.337118 16997 raft_consensus.cc:348] T 10c65e6d1cb74c90b2087b4111315e74 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.338716 16997 raft_consensus.cc:374] T 10c65e6d1cb74c90b2087b4111315e74 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.338742 16994 tablet_bootstrap.cc:492] T 10c65e6d1cb74c90b2087b4111315e74 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:33.338855 16997 raft_consensus.cc:729] T 10c65e6d1cb74c90b2087b4111315e74 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.338968 16994 ts_tablet_manager.cc:1367] T 10c65e6d1cb74c90b2087b4111315e74 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.001s
I20240503 07:59:33.339079 16997 consensus_queue.cc:260] T 10c65e6d1cb74c90b2087b4111315e74 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.339536 16997 ts_tablet_manager.cc:1398] T 10c65e6d1cb74c90b2087b4111315e74 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:33.339576 16994 raft_consensus.cc:348] T 10c65e6d1cb74c90b2087b4111315e74 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.339735 16994 raft_consensus.cc:374] T 10c65e6d1cb74c90b2087b4111315e74 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.339831 16994 raft_consensus.cc:729] T 10c65e6d1cb74c90b2087b4111315e74 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.340036 16994 consensus_queue.cc:260] T 10c65e6d1cb74c90b2087b4111315e74 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.340370 16994 ts_tablet_manager.cc:1398] T 10c65e6d1cb74c90b2087b4111315e74 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.340600 17032 ts_tablet_manager.cc:1367] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240503 07:59:33.341879 6746 catalog_manager.cc:5484] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:33.342411 17032 raft_consensus.cc:348] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.342593 17032 raft_consensus.cc:374] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.342669 17032 raft_consensus.cc:729] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.342880 17032 consensus_queue.cc:260] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.344612 17032 ts_tablet_manager.cc:1398] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:33.347050 17036 tablet_bootstrap.cc:492] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:33.347342 17036 ts_tablet_manager.cc:1367] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:33.348163 17036 raft_consensus.cc:348] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.348361 17036 raft_consensus.cc:374] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.348443 17036 raft_consensus.cc:729] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.348603 17036 consensus_queue.cc:260] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.349018 17036 ts_tablet_manager.cc:1398] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:33.390933 7261 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:33.391142 6999 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:33.395759 9351 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:33.402256 9483 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:33.409368 9615 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:33.411156 15993 raft_consensus.cc:482] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.411278 15993 raft_consensus.cc:504] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.411576 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:33.411731 15993 leader_election.cc:288] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:33.412117 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a327249bd2224ecd8754b69f228fe6d4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:33.412254 9765 raft_consensus.cc:2454] T a327249bd2224ecd8754b69f228fe6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:33.412667 6825 leader_election.cc:302] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:33.412916 15993 raft_consensus.cc:2781] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.412971 15993 raft_consensus.cc:482] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.413005 15993 raft_consensus.cc:3037] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.414227 15993 raft_consensus.cc:504] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.414628 15993 leader_election.cc:288] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:33.415630 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a327249bd2224ecd8754b69f228fe6d4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:33.415755 9505 raft_consensus.cc:3037] T a327249bd2224ecd8754b69f228fe6d4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.416855 9505 raft_consensus.cc:2454] T a327249bd2224ecd8754b69f228fe6d4 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:33.417493 6825 leader_election.cc:302] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:33.417814 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a327249bd2224ecd8754b69f228fe6d4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:33.417933 9766 raft_consensus.cc:3037] T a327249bd2224ecd8754b69f228fe6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.418992 9766 raft_consensus.cc:2454] T a327249bd2224ecd8754b69f228fe6d4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:33.419624 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a327249bd2224ecd8754b69f228fe6d4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:33.419759 9504 raft_consensus.cc:2362] T a327249bd2224ecd8754b69f228fe6d4 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1. Re-sending same reply.
I20240503 07:59:33.420943 15993 raft_consensus.cc:2781] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.421185 15993 raft_consensus.cc:686] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:33.421331 15993 consensus_queue.cc:237] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.424222 6741 catalog_manager.cc:5484] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:33.432533 6867 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:33.436371 15767 raft_consensus.cc:482] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.436528 15767 raft_consensus.cc:504] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.437114 15767 leader_election.cc:288] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:33.437544 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c65e6d1cb74c90b2087b4111315e74" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:33.437816 7278 raft_consensus.cc:2454] T 10c65e6d1cb74c90b2087b4111315e74 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:33.438333 9572 leader_election.cc:302] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:33.438629 15767 raft_consensus.cc:2781] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.438676 15767 raft_consensus.cc:482] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.438707 15767 raft_consensus.cc:3037] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.439890 15767 raft_consensus.cc:504] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.440284 15767 leader_election.cc:288] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:33.440877 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c65e6d1cb74c90b2087b4111315e74" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:33.440946 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c65e6d1cb74c90b2087b4111315e74" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:33.441020 7278 raft_consensus.cc:3037] T 10c65e6d1cb74c90b2087b4111315e74 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.441058 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c65e6d1cb74c90b2087b4111315e74" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:33.441085 9505 raft_consensus.cc:3037] T 10c65e6d1cb74c90b2087b4111315e74 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.442106 7278 raft_consensus.cc:2454] T 10c65e6d1cb74c90b2087b4111315e74 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:33.442190 9505 raft_consensus.cc:2454] T 10c65e6d1cb74c90b2087b4111315e74 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:33.442577 9572 leader_election.cc:302] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:33.444026 15767 raft_consensus.cc:2781] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.444140 15767 raft_consensus.cc:686] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:33.444272 15767 consensus_queue.cc:237] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:33.447047 6741 catalog_manager.cc:5484] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:33.447834 7129 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:33.470641 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:33.475756 7278 raft_consensus.cc:1261] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:33.476380 16132 consensus_queue.cc:1035] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.477229 9505 raft_consensus.cc:1261] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:33.478495 14889 consensus_queue.cc:1035] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.480952 6745 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet 235d98cc3a5e4913991decba34badef8 (table rebalancer_dynamic_table [id=8dd32be971c24de28551c2176484df1f]). Expected version 1 got 0
I20240503 07:59:33.485451 6745 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 235d98cc3a5e4913991decba34badef8 (table rebalancer_dynamic_table [id=8dd32be971c24de28551c2176484df1f]). Expected version 1 got 0
I20240503 07:59:33.499997 17063 tablet.cc:1722] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:33.501354 6745 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 235d98cc3a5e4913991decba34badef8 (table rebalancer_dynamic_table [id=8dd32be971c24de28551c2176484df1f]). Expected version 1 got 0
I20240503 07:59:33.502700 17063 tablet.cc:1620] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:33.510857 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=8dd32be971c24de28551c2176484df1f] alter complete (version 1)
I20240503 07:59:33.510540 17062 tablet.cc:1722] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:33.511523 17062 tablet.cc:1620] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:33.523106 17064 tablet.cc:1722] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:33.523697 17064 tablet.cc:1620] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:33.525243 6741 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet 235d98cc3a5e4913991decba34badef8 (table rebalancer_dynamic_table [id=8dd32be971c24de28551c2176484df1f]). Expected version 1 got 0
I20240503 07:59:33.534821 7278 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "64d33ff4164b49a7948aa5afee40bf3f"
dest_uuid: "5227b26243484bbfb177a995ceb50afc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54810
I20240503 07:59:33.535051 7278 raft_consensus.cc:595] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:33.601943 14839 raft_consensus.cc:482] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.602079 14839 raft_consensus.cc:504] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.602509 14839 leader_election.cc:288] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:33.602839 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab08cdb57b5649fbad1ec5792c78b33e" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:33.603001 9634 raft_consensus.cc:2454] T ab08cdb57b5649fbad1ec5792c78b33e P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:33.603360 7215 leader_election.cc:302] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:33.604192 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab08cdb57b5649fbad1ec5792c78b33e" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:33.604355 7149 raft_consensus.cc:2454] T ab08cdb57b5649fbad1ec5792c78b33e P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:33.605122 14839 raft_consensus.cc:2781] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.605197 14839 raft_consensus.cc:482] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.605233 14839 raft_consensus.cc:3037] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.606238 14839 raft_consensus.cc:504] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.606894 14839 leader_election.cc:288] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:33.606984 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab08cdb57b5649fbad1ec5792c78b33e" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:33.607117 9634 raft_consensus.cc:3037] T ab08cdb57b5649fbad1ec5792c78b33e P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.607544 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab08cdb57b5649fbad1ec5792c78b33e" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:33.607697 7149 raft_consensus.cc:3037] T ab08cdb57b5649fbad1ec5792c78b33e P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.608265 9634 raft_consensus.cc:2454] T ab08cdb57b5649fbad1ec5792c78b33e P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:33.608651 7215 leader_election.cc:302] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:33.608834 7149 raft_consensus.cc:2454] T ab08cdb57b5649fbad1ec5792c78b33e P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:33.609865 16582 raft_consensus.cc:2781] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.610073 16582 raft_consensus.cc:686] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:33.610237 16582 consensus_queue.cc:237] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:33.613185 6744 catalog_manager.cc:5484] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:33.621884 6745 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:33.632414 17062 tablet.cc:1722] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:33.632782 17062 tablet.cc:1620] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:33.634439 17063 tablet.cc:1722] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:33.634780 17063 tablet.cc:1620] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:33.638253 17064 tablet.cc:1722] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:33.638620 17064 tablet.cc:1620] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:33.644490 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=8dd32be971c24de28551c2176484df1f] alter complete (version 2)
I20240503 07:59:33.656205 17106 raft_consensus.cc:979] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc: : Instructing follower 00a3f8e159c64327832d79fb32cdcca8 to start an election
I20240503 07:59:33.656383 17106 raft_consensus.cc:1067] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Signalling peer 00a3f8e159c64327832d79fb32cdcca8 to start an election
I20240503 07:59:33.657105 7150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "64d33ff4164b49a7948aa5afee40bf3f"
dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
from {username='slave'} at 127.0.99.132:57053
I20240503 07:59:33.657305 7150 raft_consensus.cc:482] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:33.657377 7150 raft_consensus.cc:3037] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:33.658815 7150 raft_consensus.cc:504] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:33.659361 7150 leader_election.cc:288] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:33.660080 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d33ff4164b49a7948aa5afee40bf3f" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:33.660137 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d33ff4164b49a7948aa5afee40bf3f" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:33.660213 7278 raft_consensus.cc:3032] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:33.660254 7278 raft_consensus.cc:729] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:33.660288 9634 raft_consensus.cc:3037] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:33.660411 7278 consensus_queue.cc:260] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:33.660619 7278 raft_consensus.cc:3037] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:33.661335 9634 raft_consensus.cc:2454] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 2.
I20240503 07:59:33.661677 7278 raft_consensus.cc:2454] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 2.
I20240503 07:59:33.661967 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d33ff4164b49a7948aa5afee40bf3f" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:33.662104 9505 raft_consensus.cc:3037] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:33.663223 9505 raft_consensus.cc:2454] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 2.
I20240503 07:59:33.663642 7087 leader_election.cc:302] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:33.663821 16132 raft_consensus.cc:2781] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:33.663895 16132 raft_consensus.cc:686] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 2 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:33.664042 16132 consensus_queue.cc:237] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:33.667577 6746 catalog_manager.cc:5484] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 1 to 2, leader changed from 5227b26243484bbfb177a995ceb50afc (127.0.99.132) to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:33.723489 9615 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:33.724346 9351 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:33.730228 7129 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:33.737756 7261 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:33.742440 9483 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:33.755272 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:33.765756 6999 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:33.767056 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:33.768700 17062 tablet.cc:1722] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:33.776433 6742 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=8dd32be971c24de28551c2176484df1f] alter complete (version 3)
I20240503 07:59:33.776819 17064 tablet.cc:1722] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:33.783898 17063 tablet.cc:1722] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:33.788607 6867 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:33.886742 6742 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:33.886909 6742 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:33.889367 6742 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 235d98cc3a5e4913991decba34badef8
I20240503 07:59:33.890134 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 235d98cc3a5e4913991decba34badef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:33 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:33.890362 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet 235d98cc3a5e4913991decba34badef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:33 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:33.890376 9483 tablet_service.cc:1514] Processing DeleteTablet for tablet 235d98cc3a5e4913991decba34badef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:33 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:33.890677 17157 tablet_replica.cc:330] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:33.890830 17158 tablet_replica.cc:330] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:33.890859 17157 raft_consensus.cc:2229] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:33.890976 17158 raft_consensus.cc:2229] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:33.890992 17157 raft_consensus.cc:2258] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.891140 17158 raft_consensus.cc:2258] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.891712 17158 ts_tablet_manager.cc:1875] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:33.891856 17156 tablet_replica.cc:330] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:33.891954 17157 ts_tablet_manager.cc:1875] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:33.892241 17156 raft_consensus.cc:2229] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:33.892495 17156 raft_consensus.cc:2258] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.893002 17156 ts_tablet_manager.cc:1875] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:33.894410 17158 ts_tablet_manager.cc:1888] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:33.894409 17157 ts_tablet_manager.cc:1888] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:33.894490 17157 log.cc:1196] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/235d98cc3a5e4913991decba34badef8
I20240503 07:59:33.894657 17158 log.cc:1196] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/235d98cc3a5e4913991decba34badef8
I20240503 07:59:33.894902 17157 ts_tablet_manager.cc:1909] T 235d98cc3a5e4913991decba34badef8 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:33.894902 17158 ts_tablet_manager.cc:1909] T 235d98cc3a5e4913991decba34badef8 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:33.895104 17156 ts_tablet_manager.cc:1888] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:33.895264 17156 log.cc:1196] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/235d98cc3a5e4913991decba34badef8
I20240503 07:59:33.895601 17156 ts_tablet_manager.cc:1909] T 235d98cc3a5e4913991decba34badef8 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:33.896072 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 235d98cc3a5e4913991decba34badef8 (table rebalancer_dynamic_table_new_name [id=8dd32be971c24de28551c2176484df1f]) successfully deleted
I20240503 07:59:33.896294 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 235d98cc3a5e4913991decba34badef8 (table rebalancer_dynamic_table_new_name [id=8dd32be971c24de28551c2176484df1f]) successfully deleted
I20240503 07:59:33.896795 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 235d98cc3a5e4913991decba34badef8 (table rebalancer_dynamic_table_new_name [id=8dd32be971c24de28551c2176484df1f]) successfully deleted
I20240503 07:59:33.919626 15767 consensus_queue.cc:1035] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.923372 16674 consensus_queue.cc:1035] T 10c65e6d1cb74c90b2087b4111315e74 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.942672 16623 consensus_queue.cc:1035] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.947703 16623 consensus_queue.cc:1035] T a327249bd2224ecd8754b69f228fe6d4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.991019 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:34.001050 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 0727f110efdb4cf0b4077c06b48fa32a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ebb6128ed25549f7a872b4ec43846655]), partition=
I20240503 07:59:34.001058 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 0727f110efdb4cf0b4077c06b48fa32a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ebb6128ed25549f7a872b4ec43846655]), partition=
I20240503 07:59:34.001391 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0727f110efdb4cf0b4077c06b48fa32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.001394 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0727f110efdb4cf0b4077c06b48fa32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.003777 17232 tablet_bootstrap.cc:492] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:34.004014 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 0727f110efdb4cf0b4077c06b48fa32a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ebb6128ed25549f7a872b4ec43846655]), partition=
I20240503 07:59:34.004302 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0727f110efdb4cf0b4077c06b48fa32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.005307 17233 tablet_bootstrap.cc:492] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:34.005786 17232 tablet_bootstrap.cc:654] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.007141 17232 tablet_bootstrap.cc:492] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:34.007287 17233 tablet_bootstrap.cc:654] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.007365 17232 ts_tablet_manager.cc:1367] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:34.008417 17232 raft_consensus.cc:348] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.008803 17232 raft_consensus.cc:374] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.008905 17232 raft_consensus.cc:729] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.009091 17233 tablet_bootstrap.cc:492] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:34.009183 17233 ts_tablet_manager.cc:1367] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:34.009320 17232 consensus_queue.cc:260] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.009672 17232 ts_tablet_manager.cc:1398] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:34.009651 17233 raft_consensus.cc:348] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.009754 17233 raft_consensus.cc:374] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.009801 17233 raft_consensus.cc:729] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.009923 17233 consensus_queue.cc:260] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.010200 17233 ts_tablet_manager.cc:1398] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.017498 17235 tablet_bootstrap.cc:492] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:34.018812 17235 tablet_bootstrap.cc:654] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.042950 7282 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:34.044404 16132 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:34.059845 17235 tablet_bootstrap.cc:492] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:34.060047 17235 ts_tablet_manager.cc:1367] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20240503 07:59:34.060668 17235 raft_consensus.cc:348] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.060832 17235 raft_consensus.cc:374] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.060883 17235 raft_consensus.cc:729] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.061012 17235 consensus_queue.cc:260] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.061336 17235 ts_tablet_manager.cc:1398] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.090057 9615 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:34.094200 9351 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:34.097366 7261 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:34.101994 9634 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2 [term 2 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:34.102535 16132 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:34.102972 7129 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:34.107066 6999 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:34.108539 9505 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:34.109166 14889 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:34.109683 6867 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:34.123607 9745 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:34.126857 17116 consensus_queue.cc:1035] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.127488 9483 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:34.130040 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:34.152026 9351 tablet_service.cc:1467] Processing CreateTablet for tablet 61737b6ad38c470b8b7d6fceba5dbbad (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=54ae781703fb443b9b93f76b77144295]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.152312 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 61737b6ad38c470b8b7d6fceba5dbbad (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=54ae781703fb443b9b93f76b77144295]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.152442 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61737b6ad38c470b8b7d6fceba5dbbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.152621 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61737b6ad38c470b8b7d6fceba5dbbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.152601 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 61737b6ad38c470b8b7d6fceba5dbbad (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=54ae781703fb443b9b93f76b77144295]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.152670 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 28d312c7e5734833ae2ae78d140fa6d4 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=54ae781703fb443b9b93f76b77144295]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.152863 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61737b6ad38c470b8b7d6fceba5dbbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.153296 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d312c7e5734833ae2ae78d140fa6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.154675 9745 tablet_service.cc:1467] Processing CreateTablet for tablet adca01da96a74638ba00a0b24cad580f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=54ae781703fb443b9b93f76b77144295]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.154973 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adca01da96a74638ba00a0b24cad580f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.155407 6998 tablet_service.cc:1467] Processing CreateTablet for tablet adca01da96a74638ba00a0b24cad580f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=54ae781703fb443b9b93f76b77144295]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.155615 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adca01da96a74638ba00a0b24cad580f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.157752 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 28d312c7e5734833ae2ae78d140fa6d4 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=54ae781703fb443b9b93f76b77144295]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.158020 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d312c7e5734833ae2ae78d140fa6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.158061 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 28d312c7e5734833ae2ae78d140fa6d4 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=54ae781703fb443b9b93f76b77144295]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.158332 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d312c7e5734833ae2ae78d140fa6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.158778 6868 tablet_service.cc:1467] Processing CreateTablet for tablet adca01da96a74638ba00a0b24cad580f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=54ae781703fb443b9b93f76b77144295]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.159049 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adca01da96a74638ba00a0b24cad580f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.160889 17233 tablet_bootstrap.cc:492] T 28d312c7e5734833ae2ae78d140fa6d4 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:34.161312 17232 tablet_bootstrap.cc:492] T adca01da96a74638ba00a0b24cad580f P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:34.161532 17235 tablet_bootstrap.cc:492] T adca01da96a74638ba00a0b24cad580f P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:34.161939 17233 tablet_bootstrap.cc:654] T 28d312c7e5734833ae2ae78d140fa6d4 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.162443 17232 tablet_bootstrap.cc:654] T adca01da96a74638ba00a0b24cad580f P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.162955 17235 tablet_bootstrap.cc:654] T adca01da96a74638ba00a0b24cad580f P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.164116 17233 tablet_bootstrap.cc:492] T 28d312c7e5734833ae2ae78d140fa6d4 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:34.164207 17233 ts_tablet_manager.cc:1367] T 28d312c7e5734833ae2ae78d140fa6d4 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:34.164722 17233 raft_consensus.cc:348] T 28d312c7e5734833ae2ae78d140fa6d4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:34.164844 17233 raft_consensus.cc:374] T 28d312c7e5734833ae2ae78d140fa6d4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.164870 17233 raft_consensus.cc:729] T 28d312c7e5734833ae2ae78d140fa6d4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.165000 17233 consensus_queue.cc:260] T 28d312c7e5734833ae2ae78d140fa6d4 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:34.165284 17233 ts_tablet_manager.cc:1398] T 28d312c7e5734833ae2ae78d140fa6d4 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.171123 17235 tablet_bootstrap.cc:492] T adca01da96a74638ba00a0b24cad580f P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:34.171600 17235 ts_tablet_manager.cc:1367] T adca01da96a74638ba00a0b24cad580f P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20240503 07:59:34.172154 17235 raft_consensus.cc:348] T adca01da96a74638ba00a0b24cad580f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.172358 17235 raft_consensus.cc:374] T adca01da96a74638ba00a0b24cad580f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.172421 17235 raft_consensus.cc:729] T adca01da96a74638ba00a0b24cad580f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.172616 17235 consensus_queue.cc:260] T adca01da96a74638ba00a0b24cad580f P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.174780 17235 ts_tablet_manager.cc:1398] T adca01da96a74638ba00a0b24cad580f P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:34.175047 17235 tablet_bootstrap.cc:492] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:34.176247 17235 tablet_bootstrap.cc:654] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.183629 17267 tablet_bootstrap.cc:492] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:34.185462 17267 tablet_bootstrap.cc:654] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.187069 17116 consensus_queue.cc:1035] T ab08cdb57b5649fbad1ec5792c78b33e P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.187389 15993 raft_consensus.cc:482] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.187508 15993 raft_consensus.cc:504] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.187937 15993 leader_election.cc:288] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:34.188777 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0727f110efdb4cf0b4077c06b48fa32a" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:34.188938 9634 raft_consensus.cc:2454] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:34.188906 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0727f110efdb4cf0b4077c06b48fa32a" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:34.189041 9766 raft_consensus.cc:2454] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:34.189249 6822 leader_election.cc:302] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:34.189522 15993 raft_consensus.cc:2781] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.189586 15993 raft_consensus.cc:482] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.189633 15993 raft_consensus.cc:3037] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.190810 15993 raft_consensus.cc:504] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.191170 15993 leader_election.cc:288] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:34.192826 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0727f110efdb4cf0b4077c06b48fa32a" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:34.192953 9634 raft_consensus.cc:3037] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.194087 9634 raft_consensus.cc:2454] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:34.195509 6822 leader_election.cc:302] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:34.195937 17267 tablet_bootstrap.cc:492] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:34.196089 17267 ts_tablet_manager.cc:1367] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 07:59:34.196110 17235 tablet_bootstrap.cc:492] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:34.196254 17235 ts_tablet_manager.cc:1367] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20240503 07:59:34.196722 17235 raft_consensus.cc:348] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:34.196674 17267 raft_consensus.cc:348] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.196843 17267 raft_consensus.cc:374] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.196862 17235 raft_consensus.cc:374] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.196894 17235 raft_consensus.cc:729] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.196916 17267 raft_consensus.cc:729] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.197079 17235 consensus_queue.cc:260] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:34.197086 17267 consensus_queue.cc:260] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.198271 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0727f110efdb4cf0b4077c06b48fa32a" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:34.198412 9766 raft_consensus.cc:3037] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.199137 17267 ts_tablet_manager.cc:1398] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:34.199527 9766 raft_consensus.cc:2454] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:34.201171 15993 raft_consensus.cc:2781] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.201360 15993 raft_consensus.cc:686] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:34.201493 15993 consensus_queue.cc:237] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.202124 17235 ts_tablet_manager.cc:1398] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:34.203573 17266 tablet_bootstrap.cc:492] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:34.209554 17232 tablet_bootstrap.cc:492] T adca01da96a74638ba00a0b24cad580f P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:34.209414 6742 catalog_manager.cc:5484] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.209666 17232 ts_tablet_manager.cc:1367] T adca01da96a74638ba00a0b24cad580f P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.002s
I20240503 07:59:34.210170 17232 raft_consensus.cc:348] T adca01da96a74638ba00a0b24cad580f P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.210275 17232 raft_consensus.cc:374] T adca01da96a74638ba00a0b24cad580f P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.210301 17232 raft_consensus.cc:729] T adca01da96a74638ba00a0b24cad580f P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.210438 17232 consensus_queue.cc:260] T adca01da96a74638ba00a0b24cad580f P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.210722 17232 ts_tablet_manager.cc:1398] T adca01da96a74638ba00a0b24cad580f P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:34.210886 17232 tablet_bootstrap.cc:492] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:34.212100 17232 tablet_bootstrap.cc:654] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.214555 17266 tablet_bootstrap.cc:654] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.217363 17232 tablet_bootstrap.cc:492] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:34.217468 17232 ts_tablet_manager.cc:1367] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240503 07:59:34.217938 17232 raft_consensus.cc:348] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.218078 17232 raft_consensus.cc:374] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.218111 17232 raft_consensus.cc:729] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.218242 17232 consensus_queue.cc:260] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.218538 17232 ts_tablet_manager.cc:1398] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.219695 17266 tablet_bootstrap.cc:492] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:34.219815 17266 ts_tablet_manager.cc:1367] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20240503 07:59:34.220292 17266 raft_consensus.cc:348] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.220513 17266 raft_consensus.cc:374] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.220583 17266 raft_consensus.cc:729] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.220721 17266 consensus_queue.cc:260] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.221004 17266 ts_tablet_manager.cc:1398] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:34.221246 17266 tablet_bootstrap.cc:492] T 28d312c7e5734833ae2ae78d140fa6d4 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:34.227766 17266 tablet_bootstrap.cc:654] T 28d312c7e5734833ae2ae78d140fa6d4 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.229328 17266 tablet_bootstrap.cc:492] T 28d312c7e5734833ae2ae78d140fa6d4 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:34.229525 17266 ts_tablet_manager.cc:1367] T 28d312c7e5734833ae2ae78d140fa6d4 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 07:59:34.230160 17266 raft_consensus.cc:348] T 28d312c7e5734833ae2ae78d140fa6d4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:34.230320 17266 raft_consensus.cc:374] T 28d312c7e5734833ae2ae78d140fa6d4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.230397 17266 raft_consensus.cc:729] T 28d312c7e5734833ae2ae78d140fa6d4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.230535 17266 consensus_queue.cc:260] T 28d312c7e5734833ae2ae78d140fa6d4 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:34.230878 17266 ts_tablet_manager.cc:1398] T 28d312c7e5734833ae2ae78d140fa6d4 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.231077 17266 tablet_bootstrap.cc:492] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:34.232690 17266 tablet_bootstrap.cc:654] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.244762 17266 tablet_bootstrap.cc:492] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:34.244910 17266 ts_tablet_manager.cc:1367] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:34.245497 17266 raft_consensus.cc:348] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.245676 17266 raft_consensus.cc:374] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.245718 17266 raft_consensus.cc:729] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.245844 17266 consensus_queue.cc:260] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.246166 17266 ts_tablet_manager.cc:1398] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.259215 9635 consensus_queue.cc:237] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:34.260694 7150 raft_consensus.cc:1261] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:34.260737 9766 raft_consensus.cc:1261] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:34.261286 15767 consensus_queue.cc:1035] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:34.261480 16674 consensus_queue.cc:1035] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:34.264096 16674 raft_consensus.cc:2932] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:34.266069 9766 raft_consensus.cc:2932] T 730eb09f729844ce9f5952c46031c0d8 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:34.266386 7150 raft_consensus.cc:2932] T 730eb09f729844ce9f5952c46031c0d8 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:34.269208 6742 catalog_manager.cc:5484] T 730eb09f729844ce9f5952c46031c0d8 P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:34.270327 17130 raft_consensus.cc:482] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.270491 17130 raft_consensus.cc:504] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.271051 17130 leader_election.cc:288] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:34.271759 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adca01da96a74638ba00a0b24cad580f" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:34.271766 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adca01da96a74638ba00a0b24cad580f" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:34.271915 9766 raft_consensus.cc:2454] T adca01da96a74638ba00a0b24cad580f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:34.271919 6886 raft_consensus.cc:2454] T adca01da96a74638ba00a0b24cad580f P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:34.273082 9483 tablet_service.cc:1514] Processing DeleteTablet for tablet 730eb09f729844ce9f5952c46031c0d8 with delete_type TABLET_DATA_TOMBSTONED (TS cb73994515af4880b05383e78368bb4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:34.273378 17158 tablet_replica.cc:330] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:34.273530 17158 raft_consensus.cc:2229] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:34.273561 6956 leader_election.cc:302] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:34.273681 17158 raft_consensus.cc:2258] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:34.273964 17035 raft_consensus.cc:2781] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.274024 17035 raft_consensus.cc:482] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.274097 17035 raft_consensus.cc:3037] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.274158 17158 ts_tablet_manager.cc:1875] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:34.275197 17035 raft_consensus.cc:504] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.275592 17035 leader_election.cc:288] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:34.275974 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adca01da96a74638ba00a0b24cad580f" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:34.276137 9766 raft_consensus.cc:3037] T adca01da96a74638ba00a0b24cad580f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.276481 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adca01da96a74638ba00a0b24cad580f" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:34.276597 17158 ts_tablet_manager.cc:1888] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:34.276619 6886 raft_consensus.cc:3037] T adca01da96a74638ba00a0b24cad580f P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.276679 17158 log.cc:1196] T 730eb09f729844ce9f5952c46031c0d8 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/730eb09f729844ce9f5952c46031c0d8
I20240503 07:59:34.277288 9766 raft_consensus.cc:2454] T adca01da96a74638ba00a0b24cad580f P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:34.277693 6886 raft_consensus.cc:2454] T adca01da96a74638ba00a0b24cad580f P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:34.277710 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 730eb09f729844ce9f5952c46031c0d8 (table rebalancer_extra_table_26 [id=179f1c255c4d4e009e7f8119bc4e18be]) successfully deleted
I20240503 07:59:34.278134 6956 leader_election.cc:302] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:34.278333 17035 raft_consensus.cc:2781] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.278450 17035 raft_consensus.cc:686] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:34.278581 17035 consensus_queue.cc:237] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.282007 6738 catalog_manager.cc:5484] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.326926 7151 consensus_queue.cc:237] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:34.328938 7282 raft_consensus.cc:1261] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:34.329051 9357 raft_consensus.cc:1261] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 07:59:34.329152 7087 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5390e70b0449449aa3af05112bd29797. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.329387 17129 consensus_queue.cc:1035] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:34.329916 16132 consensus_queue.cc:1035] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:34.332693 17129 raft_consensus.cc:2932] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:34.333246 9357 raft_consensus.cc:2932] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:34.335469 7282 raft_consensus.cc:2932] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:34.339907 6740 catalog_manager.cc:5484] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index 5 to 6, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:34.340746 16207 raft_consensus.cc:482] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.340858 16207 raft_consensus.cc:504] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.341284 16207 leader_election.cc:288] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:34.341619 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61737b6ad38c470b8b7d6fceba5dbbad" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:34.341780 7019 raft_consensus.cc:2454] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:34.341774 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61737b6ad38c470b8b7d6fceba5dbbad" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:34.341899 6886 raft_consensus.cc:2454] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:34.342535 9309 leader_election.cc:302] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:34.342758 16207 raft_consensus.cc:2781] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.342833 16207 raft_consensus.cc:482] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.342873 16207 raft_consensus.cc:3037] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.344234 16207 raft_consensus.cc:504] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.344672 16207 leader_election.cc:288] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:34.345029 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61737b6ad38c470b8b7d6fceba5dbbad" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:34.345185 7019 raft_consensus.cc:3037] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.345579 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61737b6ad38c470b8b7d6fceba5dbbad" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:34.345724 6886 raft_consensus.cc:3037] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.346400 7019 raft_consensus.cc:2454] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:34.346748 9309 leader_election.cc:302] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:34.346904 6886 raft_consensus.cc:2454] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:34.346922 16821 raft_consensus.cc:2781] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.347066 16821 raft_consensus.cc:686] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:34.347256 16821 consensus_queue.cc:237] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.350124 6742 catalog_manager.cc:5484] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.352617 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:34.357546 9635 raft_consensus.cc:1261] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.357928 9766 raft_consensus.cc:1261] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.358299 17275 consensus_queue.cc:1035] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.358515 15993 consensus_queue.cc:1035] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.361670 6742 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 0727f110efdb4cf0b4077c06b48fa32a (table rebalancer_dynamic_table [id=ebb6128ed25549f7a872b4ec43846655]). Expected version 1 got 0
I20240503 07:59:34.362903 17356 tablet.cc:1722] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:34.363339 17356 tablet.cc:1620] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:34.363744 6738 catalog_manager.cc:5576] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683) does not have the latest schema for tablet 0727f110efdb4cf0b4077c06b48fa32a (table rebalancer_dynamic_table [id=ebb6128ed25549f7a872b4ec43846655]). Expected version 1 got 0
I20240503 07:59:34.368816 17355 tablet.cc:1722] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:34.368971 6742 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 0727f110efdb4cf0b4077c06b48fa32a (table rebalancer_dynamic_table [id=ebb6128ed25549f7a872b4ec43846655]). Expected version 1 got 0
I20240503 07:59:34.369202 17357 tablet.cc:1722] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:34.369565 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=ebb6128ed25549f7a872b4ec43846655] alter complete (version 1)
I20240503 07:59:34.370640 17357 tablet.cc:1620] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:34.370808 17355 tablet.cc:1620] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:34.382604 9635 consensus_queue.cc:237] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:34.384377 9765 raft_consensus.cc:1261] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:34.384778 9571 consensus_peers.cc:485] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 -> Peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): Couldn't send request to peer 9c085a3241f14cb7ba1332b508f4b57d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d05ebb7c3aa24804a7aa9a7e1a8e699c. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.384794 7151 raft_consensus.cc:1261] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.384899 17300 consensus_queue.cc:1035] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.385625 17300 consensus_queue.cc:1035] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.388482 17297 raft_consensus.cc:2932] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:34.389055 7151 raft_consensus.cc:2932] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:34.392803 6738 catalog_manager.cc:5484] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:34.392855 9766 raft_consensus.cc:2932] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:34.420609 9357 consensus_queue.cc:237] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:34.422039 9635 raft_consensus.cc:1261] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.422155 6886 raft_consensus.cc:1261] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:34.422220 9310 consensus_peers.cc:485] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b -> Peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): Couldn't send request to peer a372e98a670f49e9adca46ab5c836c4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a86b58fdebf942ab870d4b354414b8d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.422981 16821 consensus_queue.cc:1035] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.423372 16821 consensus_queue.cc:1035] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.425787 16821 raft_consensus.cc:2932] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:34.426240 6886 raft_consensus.cc:2932] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:34.429167 9635 raft_consensus.cc:2932] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:34.429834 6740 catalog_manager.cc:5484] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 2, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:34.468039 9765 consensus_queue.cc:237] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.469874 9357 raft_consensus.cc:1261] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.469882 9635 raft_consensus.cc:1261] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:34.470026 9700 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Couldn't send request to peer c4218904dd7e42d2be95b901a6e5db92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7009acabeee34602a3ce7462ed5a2d8a. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.470290 17402 consensus_queue.cc:1035] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.470566 16320 consensus_queue.cc:1035] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.472759 16320 raft_consensus.cc:2932] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.473199 9635 raft_consensus.cc:2932] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.474802 9374 raft_consensus.cc:2932] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.476514 6742 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:34.477769 6738 catalog_manager.cc:5484] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.482506 17355 tablet.cc:1722] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:34.482856 17355 tablet.cc:1620] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:34.484635 17356 tablet.cc:1722] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:34.484942 17356 tablet.cc:1620] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:34.486871 17357 tablet.cc:1722] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:34.487260 17357 tablet.cc:1620] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:34.490787 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=ebb6128ed25549f7a872b4ec43846655] alter complete (version 2)
I20240503 07:59:34.507858 17402 raft_consensus.cc:482] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.507999 17402 raft_consensus.cc:504] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:34.508278 7151 consensus_queue.cc:237] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:34.508486 17402 leader_election.cc:288] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:34.508875 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d312c7e5734833ae2ae78d140fa6d4" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:34.509017 7019 raft_consensus.cc:2454] T 28d312c7e5734833ae2ae78d140fa6d4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:34.509421 9702 leader_election.cc:302] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:34.509606 17401 raft_consensus.cc:2781] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.509575 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d312c7e5734833ae2ae78d140fa6d4" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:34.509655 17401 raft_consensus.cc:482] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.509693 17401 raft_consensus.cc:3037] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.509728 9635 raft_consensus.cc:2454] T 28d312c7e5734833ae2ae78d140fa6d4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:34.510900 17401 raft_consensus.cc:504] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:34.511293 17401 leader_election.cc:288] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:34.512033 9505 raft_consensus.cc:1261] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.512311 6886 raft_consensus.cc:1261] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.512356 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d312c7e5734833ae2ae78d140fa6d4" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:34.512482 7019 raft_consensus.cc:3037] T 28d312c7e5734833ae2ae78d140fa6d4 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.512567 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d312c7e5734833ae2ae78d140fa6d4" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:34.512702 9634 raft_consensus.cc:3037] T 28d312c7e5734833ae2ae78d140fa6d4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.513430 7019 raft_consensus.cc:2454] T 28d312c7e5734833ae2ae78d140fa6d4 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:34.513628 9634 raft_consensus.cc:2454] T 28d312c7e5734833ae2ae78d140fa6d4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:34.513804 9702 leader_election.cc:302] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:34.514154 17401 raft_consensus.cc:2781] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.514165 16132 consensus_queue.cc:1035] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.514250 17401 raft_consensus.cc:686] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:34.514421 17129 consensus_queue.cc:1035] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.514413 17401 consensus_queue.cc:237] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
W20240503 07:59:34.514595 7087 consensus_peers.cc:485] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): Couldn't send request to peer 5227b26243484bbfb177a995ceb50afc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79dd3d8c050d4bba829de57fe39b025f. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.516644 16132 raft_consensus.cc:2932] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:34.517119 6740 catalog_manager.cc:5484] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:34.517202 6886 raft_consensus.cc:2932] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:34.519071 9504 raft_consensus.cc:2932] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:34.520519 6742 catalog_manager.cc:5484] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:34.543551 6886 consensus_queue.cc:237] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:34.545234 7151 raft_consensus.cc:1261] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:34.545377 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5293a26b65484ae5a9e76469e644c5bc. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.545614 16623 consensus_queue.cc:1035] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:34.545943 7019 raft_consensus.cc:1261] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:34.546521 16623 consensus_queue.cc:1035] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:34.548300 17275 raft_consensus.cc:2932] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:34.548833 7019 raft_consensus.cc:2932] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:34.550843 7151 raft_consensus.cc:2932] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:34.553905 6740 catalog_manager.cc:5484] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index -1 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:34.574940 9374 consensus_queue.cc:237] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:34.576496 7019 raft_consensus.cc:1261] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.576519 9765 raft_consensus.cc:1261] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:34.576766 9310 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b -> Peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): Couldn't send request to peer 00a3f8e159c64327832d79fb32cdcca8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad1a17fdefd3425aa0cda9a8fdc827c0. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.577013 17346 consensus_queue.cc:1035] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.577463 17346 consensus_queue.cc:1035] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.579396 16821 raft_consensus.cc:2932] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:34.579916 7019 raft_consensus.cc:2932] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:34.581748 9765 raft_consensus.cc:2932] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:34.582734 6742 catalog_manager.cc:5484] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:34.595635 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:34.601797 17355 tablet.cc:1722] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:34.602218 17357 tablet.cc:1722] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:34.606115 17356 tablet.cc:1722] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:34.608656 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=ebb6128ed25549f7a872b4ec43846655] alter complete (version 3)
I20240503 07:59:34.635195 7282 consensus_queue.cc:237] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:34.636837 9635 raft_consensus.cc:1261] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:34.636937 7217 consensus_peers.cc:485] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc -> Peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): Couldn't send request to peer 9c085a3241f14cb7ba1332b508f4b57d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 350265c4f7af409481c60802a6122f7f. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.636927 9504 raft_consensus.cc:1261] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.637398 17274 consensus_queue.cc:1035] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.637639 17116 consensus_queue.cc:1035] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.640224 17274 raft_consensus.cc:2932] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:34.640722 9504 raft_consensus.cc:2932] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:34.643899 6742 catalog_manager.cc:5484] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:34.644208 9635 raft_consensus.cc:2932] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:34.669731 9374 consensus_queue.cc:237] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.671418 9765 raft_consensus.cc:1261] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.672168 17346 consensus_queue.cc:1035] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.674530 9634 raft_consensus.cc:1261] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:34.675336 9307 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b -> Peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Couldn't send request to peer c4218904dd7e42d2be95b901a6e5db92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc3367cc41234d52b74eee94d6e67069. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.675575 17346 consensus_queue.cc:1035] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.680701 17346 raft_consensus.cc:2932] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.681051 9765 raft_consensus.cc:2932] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.683086 9634 raft_consensus.cc:2932] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.685032 6742 catalog_manager.cc:5484] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.707852 7019 consensus_queue.cc:237] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:34.709605 9504 raft_consensus.cc:1261] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.709648 7282 raft_consensus.cc:1261] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:34.709782 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1bff0a8228454f95c14762536a4c82. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.710042 17517 consensus_queue.cc:1035] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.710263 17035 consensus_queue.cc:1035] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.712860 17517 raft_consensus.cc:2932] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:34.713325 9504 raft_consensus.cc:2932] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:34.716161 7282 raft_consensus.cc:2932] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:34.717098 6740 catalog_manager.cc:5484] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 2, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:34.719980 6742 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:34.720135 6742 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:34.722476 6742 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 0727f110efdb4cf0b4077c06b48fa32a
I20240503 07:59:34.723558 9747 tablet_service.cc:1514] Processing DeleteTablet for tablet 0727f110efdb4cf0b4077c06b48fa32a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:34 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:34.723562 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet 0727f110efdb4cf0b4077c06b48fa32a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:34 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:34.723871 17520 tablet_replica.cc:330] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:34.724021 17520 raft_consensus.cc:2229] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:34.724022 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet 0727f110efdb4cf0b4077c06b48fa32a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:34 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:34.724172 17520 raft_consensus.cc:2258] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:34.724313 17522 tablet_replica.cc:330] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:34.724463 17522 raft_consensus.cc:2229] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:34.724715 17522 raft_consensus.cc:2258] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:34.724807 17520 ts_tablet_manager.cc:1875] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:34.725345 17522 ts_tablet_manager.cc:1875] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:34.727512 17520 ts_tablet_manager.cc:1888] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:34.727682 17520 log.cc:1196] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/0727f110efdb4cf0b4077c06b48fa32a
I20240503 07:59:34.728036 17520 ts_tablet_manager.cc:1909] T 0727f110efdb4cf0b4077c06b48fa32a P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:34.728863 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 0727f110efdb4cf0b4077c06b48fa32a (table rebalancer_dynamic_table_new_name [id=ebb6128ed25549f7a872b4ec43846655]) successfully deleted
I20240503 07:59:34.729406 17521 tablet_replica.cc:330] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:34.729568 17522 ts_tablet_manager.cc:1888] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:34.729643 17521 raft_consensus.cc:2229] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:34.729694 17522 log.cc:1196] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0727f110efdb4cf0b4077c06b48fa32a
I20240503 07:59:34.729811 17521 raft_consensus.cc:2258] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:34.730109 17522 ts_tablet_manager.cc:1909] T 0727f110efdb4cf0b4077c06b48fa32a P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:34.730859 17521 ts_tablet_manager.cc:1875] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:34.731034 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 0727f110efdb4cf0b4077c06b48fa32a (table rebalancer_dynamic_table_new_name [id=ebb6128ed25549f7a872b4ec43846655]) successfully deleted
I20240503 07:59:34.733419 17521 ts_tablet_manager.cc:1888] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:34.733505 17521 log.cc:1196] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/0727f110efdb4cf0b4077c06b48fa32a
I20240503 07:59:34.733740 17521 ts_tablet_manager.cc:1909] T 0727f110efdb4cf0b4077c06b48fa32a P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:34.734524 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 0727f110efdb4cf0b4077c06b48fa32a (table rebalancer_dynamic_table_new_name [id=ebb6128ed25549f7a872b4ec43846655]) successfully deleted
I20240503 07:59:34.758045 17517 consensus_queue.cc:1035] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.759006 9374 consensus_queue.cc:237] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:34.760807 9765 raft_consensus.cc:1261] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.761032 7282 raft_consensus.cc:1261] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.761360 16207 consensus_queue.cc:1035] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:34.761567 9310 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b -> Peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): Couldn't send request to peer 00a3f8e159c64327832d79fb32cdcca8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 866c1bf17d444315bb3eb83a7a283b54. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.761711 16821 consensus_queue.cc:1035] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.764633 17517 consensus_queue.cc:1035] T adca01da96a74638ba00a0b24cad580f P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.764775 16207 raft_consensus.cc:2932] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:34.765197 7282 raft_consensus.cc:2932] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:34.766551 9765 raft_consensus.cc:2932] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:34.768280 6742 catalog_manager.cc:5484] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:34.793964 6888 consensus_queue.cc:237] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:34.795601 7282 raft_consensus.cc:1261] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:34.795607 7019 raft_consensus.cc:1261] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:34.796018 6822 consensus_peers.cc:485] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bbaf13cbb474734a961664ae8ebe779. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.796068 17443 consensus_queue.cc:1035] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:34.796240 16623 consensus_queue.cc:1035] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:34.798133 15993 raft_consensus.cc:2932] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:34.798563 7019 raft_consensus.cc:2932] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:34.800516 7282 raft_consensus.cc:2932] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:34.803695 6741 catalog_manager.cc:5484] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:34.806860 17562 ts_tablet_manager.cc:902] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d: Initiating tablet copy from peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:34.807606 17562 tablet_copy_client.cc:308] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:43369
I20240503 07:59:34.808693 9656 tablet_copy_service.cc:140] P 159177b3caeb434baa98e04136d050b2: Received BeginTabletCopySession request for tablet d05ebb7c3aa24804a7aa9a7e1a8e699c from peer 9c085a3241f14cb7ba1332b508f4b57d ({username='slave'} at 127.0.99.130:60049)
I20240503 07:59:34.808795 9656 tablet_copy_service.cc:161] P 159177b3caeb434baa98e04136d050b2: Beginning new tablet copy session on tablet d05ebb7c3aa24804a7aa9a7e1a8e699c from peer 9c085a3241f14cb7ba1332b508f4b57d at {username='slave'} at 127.0.99.130:60049: session id = 9c085a3241f14cb7ba1332b508f4b57d-d05ebb7c3aa24804a7aa9a7e1a8e699c
I20240503 07:59:34.809796 9656 tablet_copy_source_session.cc:205] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:34.810801 17562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05ebb7c3aa24804a7aa9a7e1a8e699c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.814184 17562 tablet_copy_client.cc:783] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:34.814535 17562 tablet_copy_client.cc:647] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:34.815640 17562 tablet_copy_client.cc:523] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:34.817159 17562 tablet_bootstrap.cc:492] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:34.821586 17562 tablet_bootstrap.cc:492] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:34.821861 17562 tablet_bootstrap.cc:492] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap complete.
I20240503 07:59:34.822054 17562 ts_tablet_manager.cc:1367] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 07:59:34.822554 17562 raft_consensus.cc:348] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:34.822677 17562 raft_consensus.cc:729] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.822813 17562 consensus_queue.cc:260] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:34.823113 17562 ts_tablet_manager.cc:1398] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:34.824024 9656 tablet_copy_service.cc:342] P 159177b3caeb434baa98e04136d050b2: Request end of tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-d05ebb7c3aa24804a7aa9a7e1a8e699c received from {username='slave'} at 127.0.99.130:60049
I20240503 07:59:34.824115 9656 tablet_copy_service.cc:433] P 159177b3caeb434baa98e04136d050b2: ending tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-d05ebb7c3aa24804a7aa9a7e1a8e699c on tablet d05ebb7c3aa24804a7aa9a7e1a8e699c with peer 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:34.824254 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:34.830262 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 5544c6ef84ad45139c80311a6f1acaec (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c9e006da56e94d33a57468bec5320ff0]), partition=
I20240503 07:59:34.830608 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5544c6ef84ad45139c80311a6f1acaec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.830585 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 5544c6ef84ad45139c80311a6f1acaec (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c9e006da56e94d33a57468bec5320ff0]), partition=
I20240503 07:59:34.830837 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5544c6ef84ad45139c80311a6f1acaec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.832762 9504 consensus_queue.cc:237] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.834466 9351 tablet_service.cc:1467] Processing CreateTablet for tablet 5544c6ef84ad45139c80311a6f1acaec (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c9e006da56e94d33a57468bec5320ff0]), partition=
I20240503 07:59:34.834823 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5544c6ef84ad45139c80311a6f1acaec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.834863 17584 tablet_bootstrap.cc:492] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:34.834935 9765 raft_consensus.cc:1261] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.835856 16461 consensus_queue.cc:1035] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:34.836019 17584 tablet_bootstrap.cc:654] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.836140 7282 raft_consensus.cc:1261] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.836853 17585 tablet_bootstrap.cc:492] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:34.837857 17585 tablet_bootstrap.cc:654] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.837567 17587 consensus_queue.cc:1035] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:34.838884 9438 consensus_peers.cc:485] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a -> Peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Couldn't send request to peer c4218904dd7e42d2be95b901a6e5db92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 287f2a8f05164e0eb722d53242e35023. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:34.839138 17592 ts_tablet_manager.cc:902] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a: Initiating tablet copy from peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:34.839170 17584 tablet_bootstrap.cc:492] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:34.839262 17584 ts_tablet_manager.cc:1367] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:34.839728 17584 raft_consensus.cc:348] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:34.839931 17584 raft_consensus.cc:374] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.839977 17590 tablet_bootstrap.cc:492] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:34.840004 17584 raft_consensus.cc:729] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.840126 17584 consensus_queue.cc:260] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:34.841423 17592 tablet_copy_client.cc:308] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37815
I20240503 07:59:34.841459 17590 tablet_bootstrap.cc:654] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.841953 9394 tablet_copy_service.cc:140] P 3befbb2d206f4be89d3e07f38837374b: Received BeginTabletCopySession request for tablet a86b58fdebf942ab870d4b354414b8d2 from peer a372e98a670f49e9adca46ab5c836c4a ({username='slave'} at 127.0.99.136:38523)
I20240503 07:59:34.842051 9394 tablet_copy_service.cc:161] P 3befbb2d206f4be89d3e07f38837374b: Beginning new tablet copy session on tablet a86b58fdebf942ab870d4b354414b8d2 from peer a372e98a670f49e9adca46ab5c836c4a at {username='slave'} at 127.0.99.136:38523: session id = a372e98a670f49e9adca46ab5c836c4a-a86b58fdebf942ab870d4b354414b8d2
I20240503 07:59:34.843259 17590 tablet_bootstrap.cc:492] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:34.843304 17584 ts_tablet_manager.cc:1398] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:34.843358 17590 ts_tablet_manager.cc:1367] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:34.843372 9394 tablet_copy_source_session.cc:205] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:34.843835 17590 raft_consensus.cc:348] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:34.843951 17590 raft_consensus.cc:374] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.843983 17590 raft_consensus.cc:729] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.844099 17590 consensus_queue.cc:260] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:34.844413 17590 ts_tablet_manager.cc:1398] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.845382 17592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86b58fdebf942ab870d4b354414b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.846312 17585 tablet_bootstrap.cc:492] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:34.846560 17585 ts_tablet_manager.cc:1367] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:34.846693 16461 raft_consensus.cc:2932] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.848085 17592 tablet_copy_client.cc:783] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:34.848121 17585 raft_consensus.cc:348] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:34.848237 17585 raft_consensus.cc:374] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.848241 17592 tablet_copy_client.cc:647] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:34.848269 17585 raft_consensus.cc:729] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.848407 17585 consensus_queue.cc:260] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:34.848771 17585 ts_tablet_manager.cc:1398] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:34.850065 7282 raft_consensus.cc:2932] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.850695 6738 catalog_manager.cc:5484] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.852110 17592 tablet_copy_client.cc:523] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:34.853014 9766 raft_consensus.cc:2932] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:34.854326 17592 tablet_bootstrap.cc:492] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:34.856918 16821 consensus_queue.cc:1035] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.865073 17592 tablet_bootstrap.cc:492] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:34.865381 17592 tablet_bootstrap.cc:492] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap complete.
I20240503 07:59:34.865578 17592 ts_tablet_manager.cc:1367] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20240503 07:59:34.866173 17592 raft_consensus.cc:348] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:34.866307 17592 raft_consensus.cc:729] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.866513 17592 consensus_queue.cc:260] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:34.866796 16207 consensus_queue.cc:1035] T 61737b6ad38c470b8b7d6fceba5dbbad P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.866972 17592 ts_tablet_manager.cc:1398] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240503 07:59:34.872310 17610 ts_tablet_manager.cc:902] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92: Initiating tablet copy from peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:34.872516 9394 tablet_copy_service.cc:342] P 3befbb2d206f4be89d3e07f38837374b: Request end of tablet copy session a372e98a670f49e9adca46ab5c836c4a-a86b58fdebf942ab870d4b354414b8d2 received from {username='slave'} at 127.0.99.136:38523
I20240503 07:59:34.872591 9394 tablet_copy_service.cc:433] P 3befbb2d206f4be89d3e07f38837374b: ending tablet copy session a372e98a670f49e9adca46ab5c836c4a-a86b58fdebf942ab870d4b354414b8d2 on tablet a86b58fdebf942ab870d4b354414b8d2 with peer a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:34.873749 17610 tablet_copy_client.cc:308] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:43683
I20240503 07:59:34.874243 9787 tablet_copy_service.cc:140] P a372e98a670f49e9adca46ab5c836c4a: Received BeginTabletCopySession request for tablet 7009acabeee34602a3ce7462ed5a2d8a from peer c4218904dd7e42d2be95b901a6e5db92 ({username='slave'} at 127.0.99.129:37889)
I20240503 07:59:34.874385 9787 tablet_copy_service.cc:161] P a372e98a670f49e9adca46ab5c836c4a: Beginning new tablet copy session on tablet 7009acabeee34602a3ce7462ed5a2d8a from peer c4218904dd7e42d2be95b901a6e5db92 at {username='slave'} at 127.0.99.129:37889: session id = c4218904dd7e42d2be95b901a6e5db92-7009acabeee34602a3ce7462ed5a2d8a
I20240503 07:59:34.875365 9787 tablet_copy_source_session.cc:205] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:34.876286 17610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7009acabeee34602a3ce7462ed5a2d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.878862 17610 tablet_copy_client.cc:783] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:34.879043 17610 tablet_copy_client.cc:647] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:34.880713 17610 tablet_copy_client.cc:523] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:34.882534 17610 tablet_bootstrap.cc:492] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:34.886027 17610 tablet_bootstrap.cc:492] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:34.886262 17610 tablet_bootstrap.cc:492] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92: Bootstrap complete.
I20240503 07:59:34.886441 17610 ts_tablet_manager.cc:1367] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 07:59:34.886844 17610 raft_consensus.cc:348] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.886958 17610 raft_consensus.cc:729] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.887079 17610 consensus_queue.cc:260] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:34.887369 17610 ts_tablet_manager.cc:1398] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.888665 9787 tablet_copy_service.cc:342] P a372e98a670f49e9adca46ab5c836c4a: Request end of tablet copy session c4218904dd7e42d2be95b901a6e5db92-7009acabeee34602a3ce7462ed5a2d8a received from {username='slave'} at 127.0.99.129:37889
I20240503 07:59:34.888758 9787 tablet_copy_service.cc:433] P a372e98a670f49e9adca46ab5c836c4a: ending tablet copy session c4218904dd7e42d2be95b901a6e5db92-7009acabeee34602a3ce7462ed5a2d8a on tablet 7009acabeee34602a3ce7462ed5a2d8a with peer c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:34.950839 17652 ts_tablet_manager.cc:902] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:34.951484 17652 tablet_copy_client.cc:308] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:34.952093 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 5390e70b0449449aa3af05112bd29797 from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:48617)
I20240503 07:59:34.952214 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 5390e70b0449449aa3af05112bd29797 from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:48617: session id = cb73994515af4880b05383e78368bb4a-5390e70b0449449aa3af05112bd29797
I20240503 07:59:34.953240 7171 tablet_copy_source_session.cc:205] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:34.954319 17652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5390e70b0449449aa3af05112bd29797. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.957973 17652 tablet_copy_client.cc:783] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:34.958238 17652 tablet_copy_client.cc:647] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:34.959509 17652 tablet_copy_client.cc:523] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:34.960988 17652 tablet_bootstrap.cc:492] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:34.993141 17652 tablet_bootstrap.cc:492] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:34.993461 17652 tablet_bootstrap.cc:492] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:34.993649 17652 ts_tablet_manager.cc:1367] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.033s user 0.023s sys 0.000s
I20240503 07:59:34.994143 17652 raft_consensus.cc:348] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:34.994271 17652 raft_consensus.cc:729] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.994405 17652 consensus_queue.cc:260] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:34.994697 17652 ts_tablet_manager.cc:1398] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:34.995558 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-5390e70b0449449aa3af05112bd29797 received from {username='slave'} at 127.0.99.134:48617
I20240503 07:59:34.995638 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session cb73994515af4880b05383e78368bb4a-5390e70b0449449aa3af05112bd29797 on tablet 5390e70b0449449aa3af05112bd29797 with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:35.023922 17402 consensus_queue.cc:1035] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.027335 17401 consensus_queue.cc:1035] T 28d312c7e5734833ae2ae78d140fa6d4 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.035908 17661 ts_tablet_manager.cc:902] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:35.036665 17661 tablet_copy_client.cc:308] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:35.037345 6908 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet 5293a26b65484ae5a9e76469e644c5bc from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:38021)
I20240503 07:59:35.037470 6908 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet 5293a26b65484ae5a9e76469e644c5bc from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:38021: session id = 3befbb2d206f4be89d3e07f38837374b-5293a26b65484ae5a9e76469e644c5bc
I20240503 07:59:35.038452 6908 tablet_copy_source_session.cc:205] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:35.039587 17661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5293a26b65484ae5a9e76469e644c5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.043118 17661 tablet_copy_client.cc:783] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:35.043308 17661 tablet_copy_client.cc:647] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:35.044833 17661 tablet_copy_client.cc:523] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:35.046214 17661 tablet_bootstrap.cc:492] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:35.053191 6747 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:35.060482 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 4c464aa1fd7949bbb88a8929012481db (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f823dc38458649d2ad8a15d1077e0a31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.060876 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c464aa1fd7949bbb88a8929012481db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.060796 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 4c464aa1fd7949bbb88a8929012481db (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f823dc38458649d2ad8a15d1077e0a31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.061038 9615 tablet_service.cc:1467] Processing CreateTablet for tablet e9ce494d3f5e419884fb3f831b6a25f6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f823dc38458649d2ad8a15d1077e0a31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.061127 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c464aa1fd7949bbb88a8929012481db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.061309 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ce494d3f5e419884fb3f831b6a25f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.061303 7129 tablet_service.cc:1467] Processing CreateTablet for tablet e9ce494d3f5e419884fb3f831b6a25f6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f823dc38458649d2ad8a15d1077e0a31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.061575 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ce494d3f5e419884fb3f831b6a25f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.063853 7261 tablet_service.cc:1467] Processing CreateTablet for tablet e9ce494d3f5e419884fb3f831b6a25f6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f823dc38458649d2ad8a15d1077e0a31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.063889 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 4c464aa1fd7949bbb88a8929012481db (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f823dc38458649d2ad8a15d1077e0a31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.064458 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ce494d3f5e419884fb3f831b6a25f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.065497 17667 tablet_bootstrap.cc:492] T 4c464aa1fd7949bbb88a8929012481db P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:35.065650 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 41d519f1a691459bb5aec6e6cdb74b5c (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f823dc38458649d2ad8a15d1077e0a31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.065912 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d519f1a691459bb5aec6e6cdb74b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.066284 17585 tablet_bootstrap.cc:492] T e9ce494d3f5e419884fb3f831b6a25f6 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:35.066529 17667 tablet_bootstrap.cc:654] T 4c464aa1fd7949bbb88a8929012481db P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.066823 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 41d519f1a691459bb5aec6e6cdb74b5c (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f823dc38458649d2ad8a15d1077e0a31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.067106 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d519f1a691459bb5aec6e6cdb74b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.067261 17585 tablet_bootstrap.cc:654] T e9ce494d3f5e419884fb3f831b6a25f6 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.068367 17585 tablet_bootstrap.cc:492] T e9ce494d3f5e419884fb3f831b6a25f6 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:35.068375 17668 tablet_bootstrap.cc:492] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:35.068480 17585 ts_tablet_manager.cc:1367] T e9ce494d3f5e419884fb3f831b6a25f6 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:35.069010 17585 raft_consensus.cc:348] T e9ce494d3f5e419884fb3f831b6a25f6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:35.069124 17585 raft_consensus.cc:374] T e9ce494d3f5e419884fb3f831b6a25f6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.069154 17585 raft_consensus.cc:729] T e9ce494d3f5e419884fb3f831b6a25f6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.069295 17585 consensus_queue.cc:260] T e9ce494d3f5e419884fb3f831b6a25f6 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:35.069599 17585 ts_tablet_manager.cc:1398] T e9ce494d3f5e419884fb3f831b6a25f6 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.069761 17668 tablet_bootstrap.cc:654] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.069864 17584 tablet_bootstrap.cc:492] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:35.070905 17584 tablet_bootstrap.cc:654] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.071564 17668 tablet_bootstrap.cc:492] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:35.071565 17667 tablet_bootstrap.cc:492] T 4c464aa1fd7949bbb88a8929012481db P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:35.071657 17668 ts_tablet_manager.cc:1367] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:35.072175 17668 raft_consensus.cc:348] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:35.072377 17668 raft_consensus.cc:374] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.072408 17668 raft_consensus.cc:729] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.072554 17668 consensus_queue.cc:260] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:35.073108 17667 ts_tablet_manager.cc:1367] T 4c464aa1fd7949bbb88a8929012481db P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:35.073877 17668 ts_tablet_manager.cc:1398] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:35.074393 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 41d519f1a691459bb5aec6e6cdb74b5c (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f823dc38458649d2ad8a15d1077e0a31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.074709 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d519f1a691459bb5aec6e6cdb74b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.074739 17584 tablet_bootstrap.cc:492] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:35.074839 17584 ts_tablet_manager.cc:1367] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:35.075253 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c464aa1fd7949bbb88a8929012481db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.075296 17661 tablet_bootstrap.cc:492] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:35.075351 17584 raft_consensus.cc:348] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:35.075459 17584 raft_consensus.cc:374] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.075486 17584 raft_consensus.cc:729] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.075606 17669 tablet_bootstrap.cc:492] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:35.075605 17584 consensus_queue.cc:260] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:35.076334 17584 ts_tablet_manager.cc:1398] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:35.076710 17669 tablet_bootstrap.cc:654] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.077566 17661 tablet_bootstrap.cc:492] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:35.077764 17661 ts_tablet_manager.cc:1367] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.004s
I20240503 07:59:35.078213 17661 raft_consensus.cc:348] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:35.078369 17661 raft_consensus.cc:729] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.078512 17661 consensus_queue.cc:260] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:35.080875 17661 ts_tablet_manager.cc:1398] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:35.079434 17667 raft_consensus.cc:348] T 4c464aa1fd7949bbb88a8929012481db P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:35.084184 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-5293a26b65484ae5a9e76469e644c5bc received from {username='slave'} at 127.0.99.133:38021
I20240503 07:59:35.084852 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-5293a26b65484ae5a9e76469e644c5bc on tablet 5293a26b65484ae5a9e76469e644c5bc with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:35.085109 17667 raft_consensus.cc:374] T 4c464aa1fd7949bbb88a8929012481db P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.085153 17667 raft_consensus.cc:729] T 4c464aa1fd7949bbb88a8929012481db P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.085263 17667 consensus_queue.cc:260] T 4c464aa1fd7949bbb88a8929012481db P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:35.085558 17667 ts_tablet_manager.cc:1398] T 4c464aa1fd7949bbb88a8929012481db P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240503 07:59:35.085764 17667 tablet_bootstrap.cc:492] T 41d519f1a691459bb5aec6e6cdb74b5c P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:35.085855 17675 tablet_bootstrap.cc:492] T 4c464aa1fd7949bbb88a8929012481db P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:35.086927 17667 tablet_bootstrap.cc:654] T 41d519f1a691459bb5aec6e6cdb74b5c P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.086923 17675 tablet_bootstrap.cc:654] T 4c464aa1fd7949bbb88a8929012481db P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.088255 17667 tablet_bootstrap.cc:492] T 41d519f1a691459bb5aec6e6cdb74b5c P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:35.088351 17667 ts_tablet_manager.cc:1367] T 41d519f1a691459bb5aec6e6cdb74b5c P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:35.088948 17667 raft_consensus.cc:348] T 41d519f1a691459bb5aec6e6cdb74b5c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:35.089059 17667 raft_consensus.cc:374] T 41d519f1a691459bb5aec6e6cdb74b5c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.089092 17667 raft_consensus.cc:729] T 41d519f1a691459bb5aec6e6cdb74b5c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.089231 17667 consensus_queue.cc:260] T 41d519f1a691459bb5aec6e6cdb74b5c P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:35.089529 17667 ts_tablet_manager.cc:1398] T 41d519f1a691459bb5aec6e6cdb74b5c P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.089594 17676 ts_tablet_manager.cc:902] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:35.090145 17676 tablet_copy_client.cc:308] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:35.090312 17669 tablet_bootstrap.cc:492] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:35.090433 17669 ts_tablet_manager.cc:1367] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:35.090837 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 79dd3d8c050d4bba829de57fe39b025f from peer 5227b26243484bbfb177a995ceb50afc ({username='slave'} at 127.0.99.132:57053)
I20240503 07:59:35.090951 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 79dd3d8c050d4bba829de57fe39b025f from peer 5227b26243484bbfb177a995ceb50afc at {username='slave'} at 127.0.99.132:57053: session id = 5227b26243484bbfb177a995ceb50afc-79dd3d8c050d4bba829de57fe39b025f
I20240503 07:59:35.091059 17669 raft_consensus.cc:348] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:35.091198 17669 raft_consensus.cc:374] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.091248 17669 raft_consensus.cc:729] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.091392 17669 consensus_queue.cc:260] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:35.091699 17669 ts_tablet_manager.cc:1398] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.091953 7171 tablet_copy_source_session.cc:205] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:35.093067 17669 tablet_bootstrap.cc:492] T 41d519f1a691459bb5aec6e6cdb74b5c P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:35.095554 17669 tablet_bootstrap.cc:654] T 41d519f1a691459bb5aec6e6cdb74b5c P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.095736 17676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dd3d8c050d4bba829de57fe39b025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.096982 17669 tablet_bootstrap.cc:492] T 41d519f1a691459bb5aec6e6cdb74b5c P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:35.096989 17675 tablet_bootstrap.cc:492] T 4c464aa1fd7949bbb88a8929012481db P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:35.097074 17675 ts_tablet_manager.cc:1367] T 4c464aa1fd7949bbb88a8929012481db P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240503 07:59:35.097064 17669 ts_tablet_manager.cc:1367] T 41d519f1a691459bb5aec6e6cdb74b5c P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:35.097484 17669 raft_consensus.cc:348] T 41d519f1a691459bb5aec6e6cdb74b5c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:35.097538 17675 raft_consensus.cc:348] T 4c464aa1fd7949bbb88a8929012481db P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:35.097594 17669 raft_consensus.cc:374] T 41d519f1a691459bb5aec6e6cdb74b5c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.097633 17675 raft_consensus.cc:374] T 4c464aa1fd7949bbb88a8929012481db P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.097648 17669 raft_consensus.cc:729] T 41d519f1a691459bb5aec6e6cdb74b5c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.097671 17675 raft_consensus.cc:729] T 4c464aa1fd7949bbb88a8929012481db P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.097784 17669 consensus_queue.cc:260] T 41d519f1a691459bb5aec6e6cdb74b5c P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:35.097807 17675 consensus_queue.cc:260] T 4c464aa1fd7949bbb88a8929012481db P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:35.098066 17669 ts_tablet_manager.cc:1398] T 41d519f1a691459bb5aec6e6cdb74b5c P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.098093 17675 ts_tablet_manager.cc:1398] T 4c464aa1fd7949bbb88a8929012481db P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:35.100239 17675 tablet_bootstrap.cc:492] T e9ce494d3f5e419884fb3f831b6a25f6 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:35.101548 17675 tablet_bootstrap.cc:654] T e9ce494d3f5e419884fb3f831b6a25f6 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.102844 17675 tablet_bootstrap.cc:492] T e9ce494d3f5e419884fb3f831b6a25f6 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:35.102936 17675 ts_tablet_manager.cc:1367] T e9ce494d3f5e419884fb3f831b6a25f6 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:35.103278 17676 tablet_copy_client.cc:783] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:35.103430 17676 tablet_copy_client.cc:647] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:35.103581 17675 raft_consensus.cc:348] T e9ce494d3f5e419884fb3f831b6a25f6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:35.103706 17675 raft_consensus.cc:374] T e9ce494d3f5e419884fb3f831b6a25f6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.103742 17675 raft_consensus.cc:729] T e9ce494d3f5e419884fb3f831b6a25f6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.103930 17675 consensus_queue.cc:260] T e9ce494d3f5e419884fb3f831b6a25f6 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:35.104319 17675 ts_tablet_manager.cc:1398] T e9ce494d3f5e419884fb3f831b6a25f6 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:35.105875 17676 tablet_copy_client.cc:523] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:35.107568 17676 tablet_bootstrap.cc:492] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:35.110817 17676 tablet_bootstrap.cc:492] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:35.110996 17676 tablet_bootstrap.cc:492] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc: Bootstrap complete.
I20240503 07:59:35.111145 17676 ts_tablet_manager.cc:1367] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:35.111532 17676 raft_consensus.cc:348] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:35.111634 17676 raft_consensus.cc:729] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.111796 17676 consensus_queue.cc:260] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:35.112085 17676 ts_tablet_manager.cc:1398] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:35.112588 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session 5227b26243484bbfb177a995ceb50afc-79dd3d8c050d4bba829de57fe39b025f received from {username='slave'} at 127.0.99.132:57053
I20240503 07:59:35.112681 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session 5227b26243484bbfb177a995ceb50afc-79dd3d8c050d4bba829de57fe39b025f on tablet 79dd3d8c050d4bba829de57fe39b025f with peer 5227b26243484bbfb177a995ceb50afc
I20240503 07:59:35.113785 17686 ts_tablet_manager.cc:902] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8: Initiating tablet copy from peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:35.114511 17686 tablet_copy_client.cc:308] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37815
I20240503 07:59:35.115029 9394 tablet_copy_service.cc:140] P 3befbb2d206f4be89d3e07f38837374b: Received BeginTabletCopySession request for tablet ad1a17fdefd3425aa0cda9a8fdc827c0 from peer 00a3f8e159c64327832d79fb32cdcca8 ({username='slave'} at 127.0.99.131:44395)
I20240503 07:59:35.115145 9394 tablet_copy_service.cc:161] P 3befbb2d206f4be89d3e07f38837374b: Beginning new tablet copy session on tablet ad1a17fdefd3425aa0cda9a8fdc827c0 from peer 00a3f8e159c64327832d79fb32cdcca8 at {username='slave'} at 127.0.99.131:44395: session id = 00a3f8e159c64327832d79fb32cdcca8-ad1a17fdefd3425aa0cda9a8fdc827c0
I20240503 07:59:35.116097 9394 tablet_copy_source_session.cc:205] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:35.117013 17686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1a17fdefd3425aa0cda9a8fdc827c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.119664 17686 tablet_copy_client.cc:783] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:35.119840 17686 tablet_copy_client.cc:647] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:35.120961 17686 tablet_copy_client.cc:523] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:35.122330 17686 tablet_bootstrap.cc:492] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:35.126502 17686 tablet_bootstrap.cc:492] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:35.126735 17686 tablet_bootstrap.cc:492] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap complete.
I20240503 07:59:35.126883 17686 ts_tablet_manager.cc:1367] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20240503 07:59:35.127341 17686 raft_consensus.cc:348] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:35.127445 17686 raft_consensus.cc:729] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.127569 17686 consensus_queue.cc:260] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:35.127840 17686 ts_tablet_manager.cc:1398] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:35.128352 9394 tablet_copy_service.cc:342] P 3befbb2d206f4be89d3e07f38837374b: Request end of tablet copy session 00a3f8e159c64327832d79fb32cdcca8-ad1a17fdefd3425aa0cda9a8fdc827c0 received from {username='slave'} at 127.0.99.131:44395
I20240503 07:59:35.128437 9394 tablet_copy_service.cc:433] P 3befbb2d206f4be89d3e07f38837374b: ending tablet copy session 00a3f8e159c64327832d79fb32cdcca8-ad1a17fdefd3425aa0cda9a8fdc827c0 on tablet ad1a17fdefd3425aa0cda9a8fdc827c0 with peer 00a3f8e159c64327832d79fb32cdcca8
I20240503 07:59:35.145576 17297 raft_consensus.cc:482] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.145789 17297 raft_consensus.cc:504] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:35.146462 17297 leader_election.cc:288] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:35.146970 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ce494d3f5e419884fb3f831b6a25f6" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:35.146999 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ce494d3f5e419884fb3f831b6a25f6" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:35.147120 7151 raft_consensus.cc:2454] T e9ce494d3f5e419884fb3f831b6a25f6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:35.147135 7282 raft_consensus.cc:2454] T e9ce494d3f5e419884fb3f831b6a25f6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:35.147493 9572 leader_election.cc:302] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2; no voters:
I20240503 07:59:35.147931 17297 raft_consensus.cc:2781] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.147992 17297 raft_consensus.cc:482] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.148027 17297 raft_consensus.cc:3037] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.148944 17297 raft_consensus.cc:504] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:35.149298 17297 leader_election.cc:288] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:35.149722 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ce494d3f5e419884fb3f831b6a25f6" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:35.149843 7282 raft_consensus.cc:3037] T e9ce494d3f5e419884fb3f831b6a25f6 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.149845 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ce494d3f5e419884fb3f831b6a25f6" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:35.149963 7151 raft_consensus.cc:3037] T e9ce494d3f5e419884fb3f831b6a25f6 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.150825 7282 raft_consensus.cc:2454] T e9ce494d3f5e419884fb3f831b6a25f6 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:35.151150 9572 leader_election.cc:302] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:35.151176 7151 raft_consensus.cc:2454] T e9ce494d3f5e419884fb3f831b6a25f6 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:35.151319 17297 raft_consensus.cc:2781] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.151388 17297 raft_consensus.cc:686] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:35.151513 17297 consensus_queue.cc:237] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:35.154103 6747 catalog_manager.cc:5484] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:35.183779 17686 ts_tablet_manager.cc:902] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8: Initiating tablet copy from peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:35.184253 17686 tablet_copy_client.cc:308] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37815
I20240503 07:59:35.184720 9394 tablet_copy_service.cc:140] P 3befbb2d206f4be89d3e07f38837374b: Received BeginTabletCopySession request for tablet 866c1bf17d444315bb3eb83a7a283b54 from peer 00a3f8e159c64327832d79fb32cdcca8 ({username='slave'} at 127.0.99.131:44395)
I20240503 07:59:35.184824 9394 tablet_copy_service.cc:161] P 3befbb2d206f4be89d3e07f38837374b: Beginning new tablet copy session on tablet 866c1bf17d444315bb3eb83a7a283b54 from peer 00a3f8e159c64327832d79fb32cdcca8 at {username='slave'} at 127.0.99.131:44395: session id = 00a3f8e159c64327832d79fb32cdcca8-866c1bf17d444315bb3eb83a7a283b54
I20240503 07:59:35.185758 9394 tablet_copy_source_session.cc:205] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:35.186712 17686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866c1bf17d444315bb3eb83a7a283b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.188796 17686 tablet_copy_client.cc:783] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:35.188967 17686 tablet_copy_client.cc:647] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:35.189869 17129 raft_consensus.cc:482] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.189956 17129 raft_consensus.cc:504] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:35.190382 17129 leader_election.cc:288] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:35.190769 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5544c6ef84ad45139c80311a6f1acaec" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:35.190913 9504 raft_consensus.cc:2454] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:35.191208 7087 leader_election.cc:302] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:35.191385 17129 raft_consensus.cc:2781] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.191430 17129 raft_consensus.cc:482] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.191462 17129 raft_consensus.cc:3037] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.191606 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5544c6ef84ad45139c80311a6f1acaec" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:35.191752 9374 raft_consensus.cc:2454] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:35.192371 17686 tablet_copy_client.cc:523] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:35.192375 17129 raft_consensus.cc:504] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:35.192809 17129 leader_election.cc:288] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:35.193143 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5544c6ef84ad45139c80311a6f1acaec" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:35.193259 9504 raft_consensus.cc:3037] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.193944 17686 tablet_bootstrap.cc:492] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:35.194252 9504 raft_consensus.cc:2454] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:35.194298 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5544c6ef84ad45139c80311a6f1acaec" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:35.194440 9374 raft_consensus.cc:3037] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.195487 9374 raft_consensus.cc:2454] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:35.196619 7087 leader_election.cc:302] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:35.196923 17129 raft_consensus.cc:2781] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.197038 17129 raft_consensus.cc:686] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:35.197171 17129 consensus_queue.cc:237] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:35.198989 17686 tablet_bootstrap.cc:492] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:35.199195 17686 tablet_bootstrap.cc:492] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap complete.
I20240503 07:59:35.199338 17686 ts_tablet_manager.cc:1367] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:35.199573 6747 catalog_manager.cc:5484] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:35.199733 17686 raft_consensus.cc:348] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:35.199829 17686 raft_consensus.cc:729] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.199961 17686 consensus_queue.cc:260] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:35.200305 17686 ts_tablet_manager.cc:1398] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:35.200819 9394 tablet_copy_service.cc:342] P 3befbb2d206f4be89d3e07f38837374b: Request end of tablet copy session 00a3f8e159c64327832d79fb32cdcca8-866c1bf17d444315bb3eb83a7a283b54 received from {username='slave'} at 127.0.99.131:44395
I20240503 07:59:35.200899 9394 tablet_copy_service.cc:433] P 3befbb2d206f4be89d3e07f38837374b: ending tablet copy session 00a3f8e159c64327832d79fb32cdcca8-866c1bf17d444315bb3eb83a7a283b54 on tablet 866c1bf17d444315bb3eb83a7a283b54 with peer 00a3f8e159c64327832d79fb32cdcca8
I20240503 07:59:35.206526 17697 ts_tablet_manager.cc:902] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:35.206950 17610 ts_tablet_manager.cc:902] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92: Initiating tablet copy from peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:35.207135 17697 tablet_copy_client.cc:308] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:35.207434 17610 tablet_copy_client.cc:308] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37815
I20240503 07:59:35.208354 9394 tablet_copy_service.cc:140] P 3befbb2d206f4be89d3e07f38837374b: Received BeginTabletCopySession request for tablet dc3367cc41234d52b74eee94d6e67069 from peer c4218904dd7e42d2be95b901a6e5db92 ({username='slave'} at 127.0.99.129:37247)
I20240503 07:59:35.208890 6908 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet 7bbaf13cbb474734a961664ae8ebe779 from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:59233)
I20240503 07:59:35.208985 6908 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet 7bbaf13cbb474734a961664ae8ebe779 from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:59233: session id = 159177b3caeb434baa98e04136d050b2-7bbaf13cbb474734a961664ae8ebe779
I20240503 07:59:35.209029 9394 tablet_copy_service.cc:161] P 3befbb2d206f4be89d3e07f38837374b: Beginning new tablet copy session on tablet dc3367cc41234d52b74eee94d6e67069 from peer c4218904dd7e42d2be95b901a6e5db92 at {username='slave'} at 127.0.99.129:37247: session id = c4218904dd7e42d2be95b901a6e5db92-dc3367cc41234d52b74eee94d6e67069
I20240503 07:59:35.210389 9394 tablet_copy_source_session.cc:205] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:35.211331 17610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3367cc41234d52b74eee94d6e67069. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.211860 6908 tablet_copy_source_session.cc:205] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:35.212790 17697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbaf13cbb474734a961664ae8ebe779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.213660 17610 tablet_copy_client.cc:783] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:35.213809 17610 tablet_copy_client.cc:647] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:35.215111 17697 tablet_copy_client.cc:783] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:35.215261 17697 tablet_copy_client.cc:647] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:35.217355 17697 tablet_copy_client.cc:523] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:35.218281 17610 tablet_copy_client.cc:523] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:35.218935 17697 tablet_bootstrap.cc:492] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:35.223716 17610 tablet_bootstrap.cc:492] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:35.227202 17610 tablet_bootstrap.cc:492] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:35.227458 17610 tablet_bootstrap.cc:492] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap complete.
I20240503 07:59:35.227624 17610 ts_tablet_manager.cc:1367] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:35.228370 17610 raft_consensus.cc:348] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:35.228502 17610 raft_consensus.cc:729] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.228619 17610 consensus_queue.cc:260] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:35.229286 17610 ts_tablet_manager.cc:1398] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.229974 9394 tablet_copy_service.cc:342] P 3befbb2d206f4be89d3e07f38837374b: Request end of tablet copy session c4218904dd7e42d2be95b901a6e5db92-dc3367cc41234d52b74eee94d6e67069 received from {username='slave'} at 127.0.99.129:37247
I20240503 07:59:35.230062 9394 tablet_copy_service.cc:433] P 3befbb2d206f4be89d3e07f38837374b: ending tablet copy session c4218904dd7e42d2be95b901a6e5db92-dc3367cc41234d52b74eee94d6e67069 on tablet dc3367cc41234d52b74eee94d6e67069 with peer c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:35.246137 17697 tablet_bootstrap.cc:492] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:35.246191 7020 raft_consensus.cc:1203] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:35.246511 17697 tablet_bootstrap.cc:492] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:35.246716 17697 ts_tablet_manager.cc:1367] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.006s
I20240503 07:59:35.247202 17697 raft_consensus.cc:348] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:35.247321 17697 raft_consensus.cc:729] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.247465 17697 consensus_queue.cc:260] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:35.248560 17697 ts_tablet_manager.cc:1398] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:35.249917 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-7bbaf13cbb474734a961664ae8ebe779 received from {username='slave'} at 127.0.99.135:59233
I20240503 07:59:35.250000 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session 159177b3caeb434baa98e04136d050b2-7bbaf13cbb474734a961664ae8ebe779 on tablet 7bbaf13cbb474734a961664ae8ebe779 with peer 159177b3caeb434baa98e04136d050b2
I20240503 07:59:35.260788 17562 ts_tablet_manager.cc:902] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:35.261423 17562 tablet_copy_client.cc:308] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:35.262099 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet 350265c4f7af409481c60802a6122f7f from peer 9c085a3241f14cb7ba1332b508f4b57d ({username='slave'} at 127.0.99.130:42749)
I20240503 07:59:35.262212 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet 350265c4f7af409481c60802a6122f7f from peer 9c085a3241f14cb7ba1332b508f4b57d at {username='slave'} at 127.0.99.130:42749: session id = 9c085a3241f14cb7ba1332b508f4b57d-350265c4f7af409481c60802a6122f7f
I20240503 07:59:35.263389 7302 tablet_copy_source_session.cc:205] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:35.264521 9615 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:35.266129 17562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350265c4f7af409481c60802a6122f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.269165 17600 raft_consensus.cc:482] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.269269 17600 raft_consensus.cc:504] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:35.269659 17600 leader_election.cc:288] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:35.269759 9765 raft_consensus.cc:1203] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:35.269887 7129 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:35.270056 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d519f1a691459bb5aec6e6cdb74b5c" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:35.270182 9634 raft_consensus.cc:2454] T 41d519f1a691459bb5aec6e6cdb74b5c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:35.270735 9438 leader_election.cc:302] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:35.271090 9483 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:35.271349 17600 raft_consensus.cc:2781] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.271406 17600 raft_consensus.cc:482] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.271440 17600 raft_consensus.cc:3037] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.272032 9747 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:35.272594 17600 raft_consensus.cc:504] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:35.272953 17600 leader_election.cc:288] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:35.273697 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d519f1a691459bb5aec6e6cdb74b5c" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:35.273819 9634 raft_consensus.cc:3037] T 41d519f1a691459bb5aec6e6cdb74b5c P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.274927 9634 raft_consensus.cc:2454] T 41d519f1a691459bb5aec6e6cdb74b5c P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:35.275184 9438 leader_election.cc:302] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:35.275349 17600 raft_consensus.cc:2781] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.275427 17600 raft_consensus.cc:686] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:35.275574 17600 consensus_queue.cc:237] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:35.278630 17562 tablet_copy_client.cc:783] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:35.278962 17562 tablet_copy_client.cc:647] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:35.279549 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d519f1a691459bb5aec6e6cdb74b5c" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:35.279700 6886 raft_consensus.cc:2454] T 41d519f1a691459bb5aec6e6cdb74b5c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:35.279553 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d519f1a691459bb5aec6e6cdb74b5c" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:35.279913 6888 raft_consensus.cc:3037] T 41d519f1a691459bb5aec6e6cdb74b5c P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.280953 6888 raft_consensus.cc:2454] T 41d519f1a691459bb5aec6e6cdb74b5c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:35.282245 6738 catalog_manager.cc:5484] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:35.285465 6867 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:35.290481 7261 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:35.301173 6998 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:35.302531 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:35.312183 17610 ts_tablet_manager.cc:902] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92: Initiating tablet copy from peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:35.312698 17610 tablet_copy_client.cc:308] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:45435
I20240503 07:59:35.313148 9525 tablet_copy_service.cc:140] P cb73994515af4880b05383e78368bb4a: Received BeginTabletCopySession request for tablet 287f2a8f05164e0eb722d53242e35023 from peer c4218904dd7e42d2be95b901a6e5db92 ({username='slave'} at 127.0.99.129:59589)
I20240503 07:59:35.313264 9525 tablet_copy_service.cc:161] P cb73994515af4880b05383e78368bb4a: Beginning new tablet copy session on tablet 287f2a8f05164e0eb722d53242e35023 from peer c4218904dd7e42d2be95b901a6e5db92 at {username='slave'} at 127.0.99.129:59589: session id = c4218904dd7e42d2be95b901a6e5db92-287f2a8f05164e0eb722d53242e35023
I20240503 07:59:35.314297 9525 tablet_copy_source_session.cc:205] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:35.315407 17610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287f2a8f05164e0eb722d53242e35023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.317991 17610 tablet_copy_client.cc:783] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:35.318173 17610 tablet_copy_client.cc:647] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:35.321081 17610 tablet_copy_client.cc:523] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:35.322722 17610 tablet_bootstrap.cc:492] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:35.323638 6747 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:35.326108 17661 ts_tablet_manager.cc:902] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:35.328117 17661 tablet_copy_client.cc:308] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46383
I20240503 07:59:35.328897 7039 tablet_copy_service.cc:140] P 9c085a3241f14cb7ba1332b508f4b57d: Received BeginTabletCopySession request for tablet fb1bff0a8228454f95c14762536a4c82 from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:33039)
I20240503 07:59:35.329010 7039 tablet_copy_service.cc:161] P 9c085a3241f14cb7ba1332b508f4b57d: Beginning new tablet copy session on tablet fb1bff0a8228454f95c14762536a4c82 from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:33039: session id = 3befbb2d206f4be89d3e07f38837374b-fb1bff0a8228454f95c14762536a4c82
I20240503 07:59:35.330032 7039 tablet_copy_source_session.cc:205] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:35.331252 17661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1bff0a8228454f95c14762536a4c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.333590 17661 tablet_copy_client.cc:783] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:35.333765 17661 tablet_copy_client.cc:647] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:35.334245 9357 raft_consensus.cc:1261] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.334590 9504 raft_consensus.cc:1261] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.335024 17129 consensus_queue.cc:1035] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.336869 17129 consensus_queue.cc:1035] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.340215 6747 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 5544c6ef84ad45139c80311a6f1acaec (table rebalancer_dynamic_table [id=c9e006da56e94d33a57468bec5320ff0]). Expected version 1 got 0
I20240503 07:59:35.340526 17562 tablet_copy_client.cc:523] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:35.341006 6744 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet 5544c6ef84ad45139c80311a6f1acaec (table rebalancer_dynamic_table [id=c9e006da56e94d33a57468bec5320ff0]). Expected version 1 got 0
I20240503 07:59:35.344369 17562 tablet_bootstrap.cc:492] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:35.349442 6744 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 5544c6ef84ad45139c80311a6f1acaec (table rebalancer_dynamic_table [id=c9e006da56e94d33a57468bec5320ff0]). Expected version 1 got 0
I20240503 07:59:35.355600 17744 tablet.cc:1722] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.355988 17661 tablet_copy_client.cc:523] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:35.357311 17744 tablet.cc:1620] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:35.357662 17661 tablet_bootstrap.cc:492] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:35.360163 17562 tablet_bootstrap.cc:492] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:35.360747 17562 tablet_bootstrap.cc:492] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap complete.
I20240503 07:59:35.360980 17562 ts_tablet_manager.cc:1367] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20240503 07:59:35.361477 17562 raft_consensus.cc:348] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:35.362306 17661 tablet_bootstrap.cc:492] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:35.362636 17661 tablet_bootstrap.cc:492] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:35.362802 17661 ts_tablet_manager.cc:1367] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 07:59:35.362448 17562 raft_consensus.cc:729] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.364238 17661 raft_consensus.cc:348] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:35.364506 17661 raft_consensus.cc:729] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.364830 17562 consensus_queue.cc:260] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:35.364708 17661 consensus_queue.cc:260] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:35.365478 17661 ts_tablet_manager.cc:1398] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:35.365478 17562 ts_tablet_manager.cc:1398] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:35.366487 7039 tablet_copy_service.cc:342] P 9c085a3241f14cb7ba1332b508f4b57d: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-fb1bff0a8228454f95c14762536a4c82 received from {username='slave'} at 127.0.99.133:33039
I20240503 07:59:35.366822 7039 tablet_copy_service.cc:433] P 9c085a3241f14cb7ba1332b508f4b57d: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-fb1bff0a8228454f95c14762536a4c82 on tablet fb1bff0a8228454f95c14762536a4c82 with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:35.368345 6747 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=c9e006da56e94d33a57468bec5320ff0] alter complete (version 1)
I20240503 07:59:35.370604 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-350265c4f7af409481c60802a6122f7f received from {username='slave'} at 127.0.99.130:42749
I20240503 07:59:35.370692 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-350265c4f7af409481c60802a6122f7f on tablet 350265c4f7af409481c60802a6122f7f with peer 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:35.383234 17747 tablet.cc:1722] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.383955 17747 tablet.cc:1620] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:35.384683 17608 mvcc.cc:205] Tried to move back new op lower bound from 7023506126161006592 to 7023506125609013248. Current Snapshot: MvccSnapshot[applied={T|T < 7023506126161006592}]
I20240503 07:59:35.385998 17610 tablet_bootstrap.cc:492] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:35.386284 17610 tablet_bootstrap.cc:492] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap complete.
I20240503 07:59:35.385944 17748 tablet.cc:1722] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.386512 17610 ts_tablet_manager.cc:1367] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.000s
I20240503 07:59:35.386699 17748 tablet.cc:1620] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:35.387042 17610 raft_consensus.cc:348] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:35.387192 17610 raft_consensus.cc:729] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.387327 17610 consensus_queue.cc:260] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:35.387377 17035 raft_consensus.cc:482] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.387485 17035 raft_consensus.cc:504] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:35.387648 17610 ts_tablet_manager.cc:1398] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.387852 17035 leader_election.cc:288] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:35.388336 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c464aa1fd7949bbb88a8929012481db" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:35.388399 6747 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet 5544c6ef84ad45139c80311a6f1acaec (table rebalancer_dynamic_table [id=c9e006da56e94d33a57468bec5320ff0]). Expected version 1 got 0
I20240503 07:59:35.388473 7282 raft_consensus.cc:2454] T 4c464aa1fd7949bbb88a8929012481db P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:35.388777 6956 leader_election.cc:302] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:35.389166 17517 raft_consensus.cc:2781] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.389221 17517 raft_consensus.cc:482] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.389256 17517 raft_consensus.cc:3037] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.389755 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c464aa1fd7949bbb88a8929012481db" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:35.389874 6888 raft_consensus.cc:2454] T 4c464aa1fd7949bbb88a8929012481db P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:35.390426 17517 raft_consensus.cc:504] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:35.390652 9525 tablet_copy_service.cc:342] P cb73994515af4880b05383e78368bb4a: Request end of tablet copy session c4218904dd7e42d2be95b901a6e5db92-287f2a8f05164e0eb722d53242e35023 received from {username='slave'} at 127.0.99.129:59589
I20240503 07:59:35.390739 9525 tablet_copy_service.cc:433] P cb73994515af4880b05383e78368bb4a: ending tablet copy session c4218904dd7e42d2be95b901a6e5db92-287f2a8f05164e0eb722d53242e35023 on tablet 287f2a8f05164e0eb722d53242e35023 with peer c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:35.390782 17517 leader_election.cc:288] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:35.393962 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c464aa1fd7949bbb88a8929012481db" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:35.393980 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c464aa1fd7949bbb88a8929012481db" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:35.394127 6886 raft_consensus.cc:3037] T 4c464aa1fd7949bbb88a8929012481db P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.394136 7282 raft_consensus.cc:3037] T 4c464aa1fd7949bbb88a8929012481db P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.395128 7282 raft_consensus.cc:2454] T 4c464aa1fd7949bbb88a8929012481db P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:35.395237 6886 raft_consensus.cc:2454] T 4c464aa1fd7949bbb88a8929012481db P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:35.395627 6956 leader_election.cc:302] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:35.397250 17035 raft_consensus.cc:2781] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.397349 17035 raft_consensus.cc:686] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:35.397500 17035 consensus_queue.cc:237] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:35.399940 6747 catalog_manager.cc:5484] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:35.430572 6886 raft_consensus.cc:1203] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:35.471223 6747 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:35.479215 17744 tablet.cc:1722] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.479625 17744 tablet.cc:1620] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:35.480463 17747 tablet.cc:1722] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.480616 17748 tablet.cc:1722] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.480810 17747 tablet.cc:1620] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:35.480919 17748 tablet.cc:1620] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:35.489300 6744 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=c9e006da56e94d33a57468bec5320ff0] alter complete (version 2)
I20240503 07:59:35.507297 9504 raft_consensus.cc:1203] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 07:59:35.565258 9615 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:35.565727 7129 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:35.567215 7261 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:35.576531 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:35.587383 9747 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:35.593293 6745 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:35.598520 6998 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:35.607334 17744 tablet.cc:1722] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:35.607739 17747 tablet.cc:1722] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:35.608551 17748 tablet.cc:1722] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:35.612533 9357 raft_consensus.cc:1203] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:35.623410 9483 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:35.625496 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=c9e006da56e94d33a57468bec5320ff0] alter complete (version 3)
I20240503 07:59:35.633989 15767 consensus_queue.cc:1035] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.634505 6867 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:35.648562 15767 consensus_queue.cc:1035] T e9ce494d3f5e419884fb3f831b6a25f6 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.650619 7278 raft_consensus.cc:1203] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:35.651139 9634 raft_consensus.cc:1203] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:35.665475 7151 raft_consensus.cc:1203] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:35.672660 7019 raft_consensus.cc:1203] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:35.715333 7151 raft_consensus.cc:1203] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:35.719780 6886 raft_consensus.cc:1203] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:35.735862 6744 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:35.736056 6744 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:35.739048 6744 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 5544c6ef84ad45139c80311a6f1acaec
I20240503 07:59:35.739727 9483 tablet_service.cc:1514] Processing DeleteTablet for tablet 5544c6ef84ad45139c80311a6f1acaec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:35 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:35.740227 9351 tablet_service.cc:1514] Processing DeleteTablet for tablet 5544c6ef84ad45139c80311a6f1acaec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:35 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:35.740492 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 5544c6ef84ad45139c80311a6f1acaec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:35 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:35.744858 17599 consensus_queue.cc:1035] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.746558 17821 tablet_replica.cc:330] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:35.746811 17821 raft_consensus.cc:2229] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:35.747040 17821 raft_consensus.cc:2258] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:35.748677 17821 ts_tablet_manager.cc:1875] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:35.751535 17821 ts_tablet_manager.cc:1888] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:35.753253 17600 consensus_queue.cc:1035] T 41d519f1a691459bb5aec6e6cdb74b5c P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.753542 17821 log.cc:1196] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5544c6ef84ad45139c80311a6f1acaec
I20240503 07:59:35.754762 17821 ts_tablet_manager.cc:1909] T 5544c6ef84ad45139c80311a6f1acaec P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:35.758330 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 5544c6ef84ad45139c80311a6f1acaec (table rebalancer_dynamic_table_new_name [id=c9e006da56e94d33a57468bec5320ff0]) successfully deleted
I20240503 07:59:35.764117 17820 tablet_replica.cc:330] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:35.764452 17820 raft_consensus.cc:2229] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:35.764680 17820 raft_consensus.cc:2258] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:35.764909 17822 tablet_replica.cc:330] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:35.765082 17822 raft_consensus.cc:2229] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:35.765298 17822 raft_consensus.cc:2258] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:35.766033 17822 ts_tablet_manager.cc:1875] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:35.767464 6888 raft_consensus.cc:1203] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:35.770006 17820 ts_tablet_manager.cc:1875] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:35.773723 17822 ts_tablet_manager.cc:1888] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:35.773905 17822 log.cc:1196] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5544c6ef84ad45139c80311a6f1acaec
I20240503 07:59:35.774322 17822 ts_tablet_manager.cc:1909] T 5544c6ef84ad45139c80311a6f1acaec P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:35.775203 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 5544c6ef84ad45139c80311a6f1acaec (table rebalancer_dynamic_table_new_name [id=c9e006da56e94d33a57468bec5320ff0]) successfully deleted
I20240503 07:59:35.782102 9357 raft_consensus.cc:1203] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:35.787525 17820 ts_tablet_manager.cc:1888] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:35.787634 17820 log.cc:1196] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/5544c6ef84ad45139c80311a6f1acaec
I20240503 07:59:35.787896 17820 ts_tablet_manager.cc:1909] T 5544c6ef84ad45139c80311a6f1acaec P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:35.789124 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 5544c6ef84ad45139c80311a6f1acaec (table rebalancer_dynamic_table_new_name [id=c9e006da56e94d33a57468bec5320ff0]) successfully deleted
I20240503 07:59:35.842854 6746 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:35.854458 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 705abc891ad24a039b0b7eaf91dd59b8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=027f1a33885b47728ab1b0a73b1b723a]), partition=
I20240503 07:59:35.854429 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 705abc891ad24a039b0b7eaf91dd59b8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=027f1a33885b47728ab1b0a73b1b723a]), partition=
I20240503 07:59:35.854748 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705abc891ad24a039b0b7eaf91dd59b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.854748 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705abc891ad24a039b0b7eaf91dd59b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.854799 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 705abc891ad24a039b0b7eaf91dd59b8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=027f1a33885b47728ab1b0a73b1b723a]), partition=
I20240503 07:59:35.855109 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705abc891ad24a039b0b7eaf91dd59b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.857776 17857 tablet_bootstrap.cc:492] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:35.860821 17858 tablet_bootstrap.cc:492] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:35.861747 17857 tablet_bootstrap.cc:654] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.861898 17858 tablet_bootstrap.cc:654] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.863096 17857 tablet_bootstrap.cc:492] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:35.863219 17857 ts_tablet_manager.cc:1367] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 07:59:35.863531 17856 tablet_bootstrap.cc:492] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:35.863708 17857 raft_consensus.cc:348] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:35.863828 17857 raft_consensus.cc:374] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.863857 17857 raft_consensus.cc:729] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.863981 17857 consensus_queue.cc:260] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:35.864274 17857 ts_tablet_manager.cc:1398] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:35.864454 17856 tablet_bootstrap.cc:654] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.874080 17856 tablet_bootstrap.cc:492] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:35.874169 17858 tablet_bootstrap.cc:492] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:35.874219 17856 ts_tablet_manager.cc:1367] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:35.874249 17858 ts_tablet_manager.cc:1367] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240503 07:59:35.874758 17858 raft_consensus.cc:348] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:35.874758 17856 raft_consensus.cc:348] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:35.874884 17858 raft_consensus.cc:374] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.874884 17856 raft_consensus.cc:374] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.874915 17856 raft_consensus.cc:729] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.874915 17858 raft_consensus.cc:729] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.875043 17856 consensus_queue.cc:260] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:35.875068 17858 consensus_queue.cc:260] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:35.875345 17856 ts_tablet_manager.cc:1398] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:35.878696 17858 ts_tablet_manager.cc:1398] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:35.889274 17035 consensus_queue.cc:1035] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.898219 17035 consensus_queue.cc:1035] T 4c464aa1fd7949bbb88a8929012481db P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.938766 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:35.949052 6745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:35.955757 9483 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:35.972632 6998 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:35.977015 9747 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:35.977944 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 6d94e0eb3ce442cfb1d5a55719b7a954 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=4e445553a2a94120b3964c609528fa41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.978307 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d94e0eb3ce442cfb1d5a55719b7a954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.979408 9747 tablet_service.cc:1467] Processing CreateTablet for tablet ce60559e643f453cb099d1d39b0175c9 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=4e445553a2a94120b3964c609528fa41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.979698 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce60559e643f453cb099d1d39b0175c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.979830 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 100a1be1cc104b8cb0412e764461f25b (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=4e445553a2a94120b3964c609528fa41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.980118 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100a1be1cc104b8cb0412e764461f25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.981237 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 6d94e0eb3ce442cfb1d5a55719b7a954 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=4e445553a2a94120b3964c609528fa41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.981501 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d94e0eb3ce442cfb1d5a55719b7a954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.981859 17857 tablet_bootstrap.cc:492] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:35.982923 17857 tablet_bootstrap.cc:654] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.983911 6998 tablet_service.cc:1467] Processing CreateTablet for tablet ce60559e643f453cb099d1d39b0175c9 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=4e445553a2a94120b3964c609528fa41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.984156 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce60559e643f453cb099d1d39b0175c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.985363 17659 raft_consensus.cc:482] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.985450 17659 raft_consensus.cc:504] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:35.985824 17659 leader_election.cc:288] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:35.986281 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705abc891ad24a039b0b7eaf91dd59b8" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:35.986438 7282 raft_consensus.cc:2454] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:35.986493 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 6d94e0eb3ce442cfb1d5a55719b7a954 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=4e445553a2a94120b3964c609528fa41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.986785 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d94e0eb3ce442cfb1d5a55719b7a954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.986786 9703 leader_election.cc:302] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:35.987107 17659 raft_consensus.cc:2781] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.987159 17659 raft_consensus.cc:482] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.987190 17659 raft_consensus.cc:3037] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.988232 17659 raft_consensus.cc:504] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:35.988601 17659 leader_election.cc:288] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:35.989223 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705abc891ad24a039b0b7eaf91dd59b8" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:35.989336 7282 raft_consensus.cc:3037] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.989511 9351 tablet_service.cc:1467] Processing CreateTablet for tablet ce60559e643f453cb099d1d39b0175c9 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=4e445553a2a94120b3964c609528fa41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.989794 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce60559e643f453cb099d1d39b0175c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.990437 7282 raft_consensus.cc:2454] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:35.990774 9703 leader_election.cc:302] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:35.990967 17659 raft_consensus.cc:2781] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.991290 17659 raft_consensus.cc:686] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:35.991433 17659 consensus_queue.cc:237] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:35.992425 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 100a1be1cc104b8cb0412e764461f25b (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=4e445553a2a94120b3964c609528fa41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.992697 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100a1be1cc104b8cb0412e764461f25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.995040 17856 tablet_bootstrap.cc:492] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:35.996084 17856 tablet_bootstrap.cc:654] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.996014 6745 catalog_manager.cc:5484] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:35.998971 17882 tablet_bootstrap.cc:492] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:36.000571 6867 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:36.001013 17882 tablet_bootstrap.cc:654] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.003171 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 100a1be1cc104b8cb0412e764461f25b (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=4e445553a2a94120b3964c609528fa41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.003484 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100a1be1cc104b8cb0412e764461f25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.004210 17857 tablet_bootstrap.cc:492] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:36.004310 17857 ts_tablet_manager.cc:1367] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.002s
I20240503 07:59:36.004856 17857 raft_consensus.cc:348] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:36.004967 17857 raft_consensus.cc:374] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.004992 17857 raft_consensus.cc:729] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.005110 17857 consensus_queue.cc:260] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:36.005384 17857 ts_tablet_manager.cc:1398] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.005918 17858 tablet_bootstrap.cc:492] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:36.006976 17858 tablet_bootstrap.cc:654] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.008379 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705abc891ad24a039b0b7eaf91dd59b8" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:36.008512 9505 raft_consensus.cc:3037] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.009577 9505 raft_consensus.cc:2454] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:36.009851 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705abc891ad24a039b0b7eaf91dd59b8" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:36.009960 9504 raft_consensus.cc:2362] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1. Re-sending same reply.
I20240503 07:59:36.010672 7129 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:36.010735 17856 tablet_bootstrap.cc:492] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:36.010833 17856 ts_tablet_manager.cc:1367] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 07:59:36.011066 9615 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:36.011350 17856 raft_consensus.cc:348] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.011458 17856 raft_consensus.cc:374] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.011484 17856 raft_consensus.cc:729] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.011598 17856 consensus_queue.cc:260] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.011868 17856 ts_tablet_manager.cc:1398] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.017130 17877 tablet_bootstrap.cc:492] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:36.017472 17880 tablet_bootstrap.cc:492] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:36.019604 17877 tablet_bootstrap.cc:654] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.027581 17880 tablet_bootstrap.cc:654] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.030289 17877 tablet_bootstrap.cc:492] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:36.030488 17877 ts_tablet_manager.cc:1367] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20240503 07:59:36.031023 17877 raft_consensus.cc:348] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.031231 17877 raft_consensus.cc:374] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.031311 17877 raft_consensus.cc:729] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.031499 17877 consensus_queue.cc:260] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.031874 17877 ts_tablet_manager.cc:1398] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:36.059253 17858 tablet_bootstrap.cc:492] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:36.059382 17858 ts_tablet_manager.cc:1367] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.002s
I20240503 07:59:36.059880 17858 raft_consensus.cc:348] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.059998 17858 raft_consensus.cc:374] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.060022 17858 raft_consensus.cc:729] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.060171 17858 consensus_queue.cc:260] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.060463 17858 ts_tablet_manager.cc:1398] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:36.062429 17878 tablet_bootstrap.cc:492] T 100a1be1cc104b8cb0412e764461f25b P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:36.066267 17882 tablet_bootstrap.cc:492] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:36.066427 17882 ts_tablet_manager.cc:1367] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.000s
I20240503 07:59:36.066426 7261 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:36.066958 17878 tablet_bootstrap.cc:654] T 100a1be1cc104b8cb0412e764461f25b P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.066926 17882 raft_consensus.cc:348] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:36.067049 17882 raft_consensus.cc:374] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.067077 17882 raft_consensus.cc:729] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.067189 17880 tablet_bootstrap.cc:492] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:36.067219 17882 consensus_queue.cc:260] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:36.067283 17880 ts_tablet_manager.cc:1367] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20240503 07:59:36.067540 17882 ts_tablet_manager.cc:1398] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.067801 17882 tablet_bootstrap.cc:492] T 100a1be1cc104b8cb0412e764461f25b P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:36.067749 17880 raft_consensus.cc:348] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.067850 17880 raft_consensus.cc:374] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.067878 17880 raft_consensus.cc:729] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.067989 17880 consensus_queue.cc:260] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.068360 17880 ts_tablet_manager.cc:1398] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.068814 17882 tablet_bootstrap.cc:654] T 100a1be1cc104b8cb0412e764461f25b P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.070118 17882 tablet_bootstrap.cc:492] T 100a1be1cc104b8cb0412e764461f25b P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:36.070243 17882 ts_tablet_manager.cc:1367] T 100a1be1cc104b8cb0412e764461f25b P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:36.070722 17882 raft_consensus.cc:348] T 100a1be1cc104b8cb0412e764461f25b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.070914 17882 raft_consensus.cc:374] T 100a1be1cc104b8cb0412e764461f25b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.070981 17882 raft_consensus.cc:729] T 100a1be1cc104b8cb0412e764461f25b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.071565 17882 consensus_queue.cc:260] T 100a1be1cc104b8cb0412e764461f25b P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.071962 17882 ts_tablet_manager.cc:1398] T 100a1be1cc104b8cb0412e764461f25b P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:36.077474 17659 raft_consensus.cc:482] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.077601 17659 raft_consensus.cc:504] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:36.077987 17659 leader_election.cc:288] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:36.078337 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce60559e643f453cb099d1d39b0175c9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:36.078475 9373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce60559e643f453cb099d1d39b0175c9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:36.078608 9373 raft_consensus.cc:2454] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
W20240503 07:59:36.078806 9702 leader_election.cc:341] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:36.078944 9701 leader_election.cc:302] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters: 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:36.079185 17659 raft_consensus.cc:2781] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.079234 17659 raft_consensus.cc:482] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.079268 17659 raft_consensus.cc:3037] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.080448 17659 raft_consensus.cc:504] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:36.080819 17659 leader_election.cc:288] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:36.081403 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce60559e643f453cb099d1d39b0175c9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:36.081503 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce60559e643f453cb099d1d39b0175c9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:36.081617 9374 raft_consensus.cc:3037] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:36.081828 9702 leader_election.cc:341] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:36.082521 9374 raft_consensus.cc:2454] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:36.082834 9701 leader_election.cc:302] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a; no voters: 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:36.083086 17659 raft_consensus.cc:2781] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.083151 17659 raft_consensus.cc:686] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:36.083256 17878 tablet_bootstrap.cc:492] T 100a1be1cc104b8cb0412e764461f25b P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:36.083271 17659 consensus_queue.cc:237] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:36.083359 17878 ts_tablet_manager.cc:1367] T 100a1be1cc104b8cb0412e764461f25b P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 07:59:36.083724 17876 raft_consensus.cc:482] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.083822 17876 raft_consensus.cc:504] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.084741 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d94e0eb3ce442cfb1d5a55719b7a954" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:36.084817 17876 leader_election.cc:288] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:36.084858 7278 raft_consensus.cc:2454] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:36.085371 9572 leader_election.cc:302] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:36.085600 17878 raft_consensus.cc:348] T 100a1be1cc104b8cb0412e764461f25b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.085703 17878 raft_consensus.cc:374] T 100a1be1cc104b8cb0412e764461f25b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.085741 17878 raft_consensus.cc:729] T 100a1be1cc104b8cb0412e764461f25b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.085880 17876 raft_consensus.cc:2781] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.085933 17876 raft_consensus.cc:482] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.085964 17876 raft_consensus.cc:3037] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.086530 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d94e0eb3ce442cfb1d5a55719b7a954" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:36.086670 7150 raft_consensus.cc:2454] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:36.087249 6745 catalog_manager.cc:5484] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:36.088194 17876 raft_consensus.cc:504] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.088838 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d94e0eb3ce442cfb1d5a55719b7a954" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:36.088963 7278 raft_consensus.cc:3037] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.089980 7278 raft_consensus.cc:2454] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:36.090291 9572 leader_election.cc:302] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:36.090484 17297 raft_consensus.cc:2781] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.090556 17297 raft_consensus.cc:686] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:36.090685 17297 consensus_queue.cc:237] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.093052 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d94e0eb3ce442cfb1d5a55719b7a954" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:36.093199 7150 raft_consensus.cc:3037] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.094300 7150 raft_consensus.cc:2454] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:36.094765 17876 leader_election.cc:288] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:36.094653 6745 catalog_manager.cc:5484] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:36.095024 17878 consensus_queue.cc:260] T 100a1be1cc104b8cb0412e764461f25b P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.095500 17878 ts_tablet_manager.cc:1398] T 100a1be1cc104b8cb0412e764461f25b P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240503 07:59:36.095851 17878 tablet_bootstrap.cc:492] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:36.097257 17878 tablet_bootstrap.cc:654] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.103552 17878 tablet_bootstrap.cc:492] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:36.103832 17878 ts_tablet_manager.cc:1367] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:36.104401 17878 raft_consensus.cc:348] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:36.104607 17878 raft_consensus.cc:374] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.104684 17878 raft_consensus.cc:729] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.104893 17878 consensus_queue.cc:260] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:36.105324 17878 ts_tablet_manager.cc:1398] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.116178 6742 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:36.121091 9504 raft_consensus.cc:1261] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.121158 7278 raft_consensus.cc:1261] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.121524 17659 consensus_queue.cc:1035] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.121659 17401 consensus_queue.cc:1035] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.124982 6742 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 705abc891ad24a039b0b7eaf91dd59b8 (table rebalancer_dynamic_table [id=027f1a33885b47728ab1b0a73b1b723a]). Expected version 1 got 0
I20240503 07:59:36.125268 17901 tablet.cc:1722] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:36.125635 17901 tablet.cc:1620] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:36.125975 6745 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet 705abc891ad24a039b0b7eaf91dd59b8 (table rebalancer_dynamic_table [id=027f1a33885b47728ab1b0a73b1b723a]). Expected version 1 got 0
I20240503 07:59:36.128322 17902 tablet.cc:1722] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:36.128649 17902 tablet.cc:1620] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:36.130293 6742 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 705abc891ad24a039b0b7eaf91dd59b8 (table rebalancer_dynamic_table [id=027f1a33885b47728ab1b0a73b1b723a]). Expected version 1 got 0
I20240503 07:59:36.130869 17903 tablet.cc:1722] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:36.131260 17903 tablet.cc:1620] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:36.133970 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=027f1a33885b47728ab1b0a73b1b723a] alter complete (version 1)
I20240503 07:59:36.150152 7019 consensus_queue.cc:237] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:36.151494 9374 raft_consensus.cc:1261] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:36.151513 7278 raft_consensus.cc:1261] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:36.152020 17035 consensus_queue.cc:1035] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:36.152416 17517 consensus_queue.cc:1035] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:36.153992 17517 raft_consensus.cc:2932] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:36.154754 9374 raft_consensus.cc:2932] T fb1bff0a8228454f95c14762536a4c82 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:36.155153 7278 raft_consensus.cc:2932] T fb1bff0a8228454f95c14762536a4c82 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:36.157223 6738 catalog_manager.cc:5484] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:36.161041 9483 tablet_service.cc:1514] Processing DeleteTablet for tablet fb1bff0a8228454f95c14762536a4c82 with delete_type TABLET_DATA_TOMBSTONED (TS cb73994515af4880b05383e78368bb4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:36.161327 17820 tablet_replica.cc:330] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:36.161500 17820 raft_consensus.cc:2229] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.161640 17820 raft_consensus.cc:2258] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.162135 17820 ts_tablet_manager.cc:1875] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:36.164192 17820 ts_tablet_manager.cc:1888] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:36.164273 17820 log.cc:1196] T fb1bff0a8228454f95c14762536a4c82 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/fb1bff0a8228454f95c14762536a4c82
I20240503 07:59:36.165784 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet fb1bff0a8228454f95c14762536a4c82 (table rebalancer_extra_table_26 [id=179f1c255c4d4e009e7f8119bc4e18be]) successfully deleted
I20240503 07:59:36.236496 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:36.242853 17901 tablet.cc:1722] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:36.243233 17901 tablet.cc:1620] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:36.244421 17903 tablet.cc:1722] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:36.244746 17903 tablet.cc:1620] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:36.246860 17902 tablet.cc:1722] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:36.247184 17902 tablet.cc:1620] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:36.253166 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=027f1a33885b47728ab1b0a73b1b723a] alter complete (version 2)
I20240503 07:59:36.254051 17599 raft_consensus.cc:482] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.254165 17599 raft_consensus.cc:504] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.254688 17599 leader_election.cc:288] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:36.255148 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100a1be1cc104b8cb0412e764461f25b" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:36.255317 9374 raft_consensus.cc:2454] T 100a1be1cc104b8cb0412e764461f25b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:36.255628 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100a1be1cc104b8cb0412e764461f25b" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:36.255779 7020 raft_consensus.cc:2454] T 100a1be1cc104b8cb0412e764461f25b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:36.256204 9439 leader_election.cc:302] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:36.256865 17599 raft_consensus.cc:2781] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.256929 17599 raft_consensus.cc:482] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.256966 17599 raft_consensus.cc:3037] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.257863 17599 raft_consensus.cc:504] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.258214 17599 leader_election.cc:288] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:36.258574 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100a1be1cc104b8cb0412e764461f25b" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:36.258700 7020 raft_consensus.cc:3037] T 100a1be1cc104b8cb0412e764461f25b P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.259846 7020 raft_consensus.cc:2454] T 100a1be1cc104b8cb0412e764461f25b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:36.262315 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100a1be1cc104b8cb0412e764461f25b" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:36.262486 9374 raft_consensus.cc:3037] T 100a1be1cc104b8cb0412e764461f25b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.263715 9374 raft_consensus.cc:2454] T 100a1be1cc104b8cb0412e764461f25b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:36.264588 9439 leader_election.cc:302] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:36.265441 17599 raft_consensus.cc:2781] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.265532 17599 raft_consensus.cc:686] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:36.265666 17599 consensus_queue.cc:237] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.268656 6741 catalog_manager.cc:5484] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:36.326514 7129 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:36.329233 9350 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:36.330803 7000 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:36.332362 9615 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:36.340989 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:36.342053 9483 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:36.343456 7261 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:36.346606 17517 raft_consensus.cc:482] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.346725 17517 raft_consensus.cc:504] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:36.347121 17517 leader_election.cc:288] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:36.349071 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce60559e643f453cb099d1d39b0175c9" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:36.353540 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce60559e643f453cb099d1d39b0175c9" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:36.353744 9374 raft_consensus.cc:2379] T ce60559e643f453cb099d1d39b0175c9 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c085a3241f14cb7ba1332b508f4b57d in current term 1: Already voted for candidate a372e98a670f49e9adca46ab5c836c4a in this term.
I20240503 07:59:36.354259 6954 leader_election.cc:302] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d; no voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:36.354789 17517 raft_consensus.cc:3037] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.355728 17517 raft_consensus.cc:2726] T ce60559e643f453cb099d1d39b0175c9 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:36.360945 6867 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:36.366276 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:36.373313 17901 tablet.cc:1722] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.373747 17902 tablet.cc:1722] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.389815 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=027f1a33885b47728ab1b0a73b1b723a] alter complete (version 3)
I20240503 07:59:36.397073 17903 tablet.cc:1722] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.470567 7278 consensus_queue.cc:237] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:36.471973 9504 raft_consensus.cc:1261] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:36.472133 7020 raft_consensus.cc:1261] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:36.472432 17480 consensus_queue.cc:1035] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:36.472666 17116 consensus_queue.cc:1035] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:36.474674 17989 raft_consensus.cc:2932] T 350265c4f7af409481c60802a6122f7f P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:36.474978 7019 raft_consensus.cc:2932] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:36.475066 9504 raft_consensus.cc:2932] T 350265c4f7af409481c60802a6122f7f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:36.478849 6741 catalog_manager.cc:5484] T 350265c4f7af409481c60802a6122f7f P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:36.482252 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet 350265c4f7af409481c60802a6122f7f with delete_type TABLET_DATA_TOMBSTONED (TS 159177b3caeb434baa98e04136d050b2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:36.483106 17995 tablet_replica.cc:330] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:36.483603 17995 raft_consensus.cc:2229] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.483776 17995 raft_consensus.cc:2258] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.484387 17995 ts_tablet_manager.cc:1875] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:36.486819 17995 ts_tablet_manager.cc:1888] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:36.486901 17995 log.cc:1196] T 350265c4f7af409481c60802a6122f7f P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/350265c4f7af409481c60802a6122f7f
I20240503 07:59:36.487566 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 350265c4f7af409481c60802a6122f7f (table rebalancer_extra_table_18 [id=fb966e884afa4325a2dc1798eb47fbf4]) successfully deleted
I20240503 07:59:36.494661 6746 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:36.494824 6746 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:36.497800 6746 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 705abc891ad24a039b0b7eaf91dd59b8
I20240503 07:59:36.498324 9747 tablet_service.cc:1514] Processing DeleteTablet for tablet 705abc891ad24a039b0b7eaf91dd59b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:36.498447 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet 705abc891ad24a039b0b7eaf91dd59b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:36.498664 9483 tablet_service.cc:1514] Processing DeleteTablet for tablet 705abc891ad24a039b0b7eaf91dd59b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:36.498792 18003 tablet_replica.cc:330] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:36.498852 17820 tablet_replica.cc:330] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:36.498739 18002 tablet_replica.cc:330] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:36.498953 18003 raft_consensus.cc:2229] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.498991 17820 raft_consensus.cc:2229] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.498998 18002 raft_consensus.cc:2229] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.499083 18003 raft_consensus.cc:2258] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.499118 17820 raft_consensus.cc:2258] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.499230 18002 raft_consensus.cc:2258] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.499856 18002 ts_tablet_manager.cc:1875] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.499989 17820 ts_tablet_manager.cc:1875] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.503502 18002 ts_tablet_manager.cc:1888] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.503569 17820 ts_tablet_manager.cc:1888] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.503638 18002 log.cc:1196] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/705abc891ad24a039b0b7eaf91dd59b8
I20240503 07:59:36.503895 17820 log.cc:1196] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/705abc891ad24a039b0b7eaf91dd59b8
I20240503 07:59:36.503942 18002 ts_tablet_manager.cc:1909] T 705abc891ad24a039b0b7eaf91dd59b8 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:36.504151 17820 ts_tablet_manager.cc:1909] T 705abc891ad24a039b0b7eaf91dd59b8 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:36.504805 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 705abc891ad24a039b0b7eaf91dd59b8 (table rebalancer_dynamic_table_new_name [id=027f1a33885b47728ab1b0a73b1b723a]) successfully deleted
I20240503 07:59:36.505002 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 705abc891ad24a039b0b7eaf91dd59b8 (table rebalancer_dynamic_table_new_name [id=027f1a33885b47728ab1b0a73b1b723a]) successfully deleted
I20240503 07:59:36.505081 18003 ts_tablet_manager.cc:1875] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.507659 18003 ts_tablet_manager.cc:1888] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.507741 18003 log.cc:1196] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/705abc891ad24a039b0b7eaf91dd59b8
I20240503 07:59:36.508016 18003 ts_tablet_manager.cc:1909] T 705abc891ad24a039b0b7eaf91dd59b8 P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:36.508810 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 705abc891ad24a039b0b7eaf91dd59b8 (table rebalancer_dynamic_table_new_name [id=027f1a33885b47728ab1b0a73b1b723a]) successfully deleted
I20240503 07:59:36.520741 15767 consensus_queue.cc:1035] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.524943 17297 consensus_queue.cc:1035] T 6d94e0eb3ce442cfb1d5a55719b7a954 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.601074 6744 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:36.603062 7129 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:36.604418 9615 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:36.608544 6867 tablet_service.cc:1467] Processing CreateTablet for tablet c49c7188a3ee46b19ab45a469c94f662 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9f360a15835845a7a34f04b2b22a8e2b]), partition=
I20240503 07:59:36.608762 7129 tablet_service.cc:1467] Processing CreateTablet for tablet c49c7188a3ee46b19ab45a469c94f662 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9f360a15835845a7a34f04b2b22a8e2b]), partition=
I20240503 07:59:36.608830 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49c7188a3ee46b19ab45a469c94f662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.609040 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49c7188a3ee46b19ab45a469c94f662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.610584 9483 tablet_service.cc:1467] Processing CreateTablet for tablet c49c7188a3ee46b19ab45a469c94f662 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9f360a15835845a7a34f04b2b22a8e2b]), partition=
I20240503 07:59:36.610824 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49c7188a3ee46b19ab45a469c94f662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.612138 18049 tablet_bootstrap.cc:492] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:36.612381 17659 consensus_queue.cc:1035] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.614212 18049 tablet_bootstrap.cc:654] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.620074 17401 consensus_queue.cc:1035] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.624059 18049 tablet_bootstrap.cc:492] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:36.624189 18049 ts_tablet_manager.cc:1367] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:36.624743 18049 raft_consensus.cc:348] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.624882 18049 raft_consensus.cc:374] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.624917 18049 raft_consensus.cc:729] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.625058 18049 consensus_queue.cc:260] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.625402 18049 ts_tablet_manager.cc:1398] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.626255 9483 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:36.627568 9747 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:36.630307 9350 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:36.633934 7000 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:36.642490 18051 tablet_bootstrap.cc:492] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:36.643462 18050 tablet_bootstrap.cc:492] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:36.644591 18050 tablet_bootstrap.cc:654] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.645272 18051 tablet_bootstrap.cc:654] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.648526 18051 tablet_bootstrap.cc:492] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:36.648660 18051 ts_tablet_manager.cc:1367] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:36.649152 18051 raft_consensus.cc:348] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.649276 18051 raft_consensus.cc:374] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.649300 18051 raft_consensus.cc:729] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.649423 18051 consensus_queue.cc:260] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.649739 18051 ts_tablet_manager.cc:1398] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.650203 6867 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:36.680282 7261 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:36.696355 18050 tablet_bootstrap.cc:492] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:36.696480 18050 ts_tablet_manager.cc:1367] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.000s
I20240503 07:59:36.697016 18050 raft_consensus.cc:348] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.697173 18050 raft_consensus.cc:374] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.697206 18050 raft_consensus.cc:729] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.697345 18050 consensus_queue.cc:260] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.697643 18050 ts_tablet_manager.cc:1398] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.710945 17599 consensus_queue.cc:1035] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.717300 17600 consensus_queue.cc:1035] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.719022 15993 raft_consensus.cc:482] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.719163 15993 raft_consensus.cc:504] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.719576 15993 leader_election.cc:288] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:36.722417 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49c7188a3ee46b19ab45a469c94f662" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:36.722579 7151 raft_consensus.cc:2454] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:36.722978 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49c7188a3ee46b19ab45a469c94f662" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:36.723119 9504 raft_consensus.cc:2454] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 0.
I20240503 07:59:36.723147 6825 leader_election.cc:302] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:36.723827 15993 raft_consensus.cc:2781] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.723894 15993 raft_consensus.cc:482] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.723927 15993 raft_consensus.cc:3037] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.724997 15993 raft_consensus.cc:504] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.725356 15993 leader_election.cc:288] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:36.725786 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49c7188a3ee46b19ab45a469c94f662" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:36.725920 7151 raft_consensus.cc:3037] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.726650 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49c7188a3ee46b19ab45a469c94f662" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:36.726778 9504 raft_consensus.cc:3037] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.727037 7151 raft_consensus.cc:2454] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:36.727530 6825 leader_election.cc:302] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:36.727701 15993 raft_consensus.cc:2781] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.727775 15993 raft_consensus.cc:686] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:36.727838 9504 raft_consensus.cc:2454] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:36.727912 15993 consensus_queue.cc:237] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:36.730481 6741 catalog_manager.cc:5484] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:36.758088 9374 consensus_queue.cc:237] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:36.759618 7019 raft_consensus.cc:1261] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:36.759804 7151 raft_consensus.cc:1261] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:36.760046 16821 consensus_queue.cc:1035] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:36.760241 17499 consensus_queue.cc:1035] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:36.762699 7151 raft_consensus.cc:2932] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:36.762745 7020 raft_consensus.cc:2932] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:36.762271 16821 raft_consensus.cc:2932] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:36.766670 6744 catalog_manager.cc:5484] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:36.769692 9747 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1a17fdefd3425aa0cda9a8fdc827c0 with delete_type TABLET_DATA_TOMBSTONED (TS a372e98a670f49e9adca46ab5c836c4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:36.769968 18002 tablet_replica.cc:330] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:36.770123 18002 raft_consensus.cc:2229] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.770229 18002 raft_consensus.cc:2258] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.770716 18002 ts_tablet_manager.cc:1875] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:36.772629 18002 ts_tablet_manager.cc:1888] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:36.772701 18002 log.cc:1196] T ad1a17fdefd3425aa0cda9a8fdc827c0 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/ad1a17fdefd3425aa0cda9a8fdc827c0
I20240503 07:59:36.773312 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet ad1a17fdefd3425aa0cda9a8fdc827c0 (table rebalancer_extra_table_24 [id=7055be84cc0f4e368d98a9e128f121af]) successfully deleted
I20240503 07:59:36.776726 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:36.784005 9747 tablet_service.cc:1467] Processing CreateTablet for tablet cc2d83ce05384dc9b3e306569fa200b3 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=3eb8fcee8b36492886e159a251f6a824]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.784356 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2d83ce05384dc9b3e306569fa200b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.784646 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 197241807308405baed79aaf07c25fa4 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=3eb8fcee8b36492886e159a251f6a824]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.784824 9484 tablet_service.cc:1467] Processing CreateTablet for tablet 5bec006637ea4caebf1ebe64056c7a26 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=3eb8fcee8b36492886e159a251f6a824]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.784899 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197241807308405baed79aaf07c25fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.785063 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bec006637ea4caebf1ebe64056c7a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.786865 18091 tablet_bootstrap.cc:492] T cc2d83ce05384dc9b3e306569fa200b3 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:36.787036 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 197241807308405baed79aaf07c25fa4 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=3eb8fcee8b36492886e159a251f6a824]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.787292 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197241807308405baed79aaf07c25fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.787314 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 5bec006637ea4caebf1ebe64056c7a26 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=3eb8fcee8b36492886e159a251f6a824]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.787562 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bec006637ea4caebf1ebe64056c7a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.788336 18091 tablet_bootstrap.cc:654] T cc2d83ce05384dc9b3e306569fa200b3 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.789311 18091 tablet_bootstrap.cc:492] T cc2d83ce05384dc9b3e306569fa200b3 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:36.789376 18091 ts_tablet_manager.cc:1367] T cc2d83ce05384dc9b3e306569fa200b3 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:36.789754 18051 tablet_bootstrap.cc:492] T 5bec006637ea4caebf1ebe64056c7a26 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:36.790103 18091 raft_consensus.cc:348] T cc2d83ce05384dc9b3e306569fa200b3 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:36.790335 18091 raft_consensus.cc:374] T cc2d83ce05384dc9b3e306569fa200b3 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.790488 18091 raft_consensus.cc:729] T cc2d83ce05384dc9b3e306569fa200b3 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.790753 18051 tablet_bootstrap.cc:654] T 5bec006637ea4caebf1ebe64056c7a26 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.791174 18091 consensus_queue.cc:260] T cc2d83ce05384dc9b3e306569fa200b3 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:36.791704 18050 tablet_bootstrap.cc:492] T 5bec006637ea4caebf1ebe64056c7a26 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:36.792639 18050 tablet_bootstrap.cc:654] T 5bec006637ea4caebf1ebe64056c7a26 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.793267 18091 ts_tablet_manager.cc:1398] T cc2d83ce05384dc9b3e306569fa200b3 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:36.793483 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 197241807308405baed79aaf07c25fa4 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=3eb8fcee8b36492886e159a251f6a824]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.793792 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197241807308405baed79aaf07c25fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.795822 18051 tablet_bootstrap.cc:492] T 5bec006637ea4caebf1ebe64056c7a26 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:36.795949 18051 ts_tablet_manager.cc:1367] T 5bec006637ea4caebf1ebe64056c7a26 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:36.796470 18051 raft_consensus.cc:348] T 5bec006637ea4caebf1ebe64056c7a26 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:36.796587 18051 raft_consensus.cc:374] T 5bec006637ea4caebf1ebe64056c7a26 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.796617 18051 raft_consensus.cc:729] T 5bec006637ea4caebf1ebe64056c7a26 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.796664 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 5bec006637ea4caebf1ebe64056c7a26 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=3eb8fcee8b36492886e159a251f6a824]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.796821 18051 consensus_queue.cc:260] T 5bec006637ea4caebf1ebe64056c7a26 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:36.796918 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bec006637ea4caebf1ebe64056c7a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.797158 18097 tablet_bootstrap.cc:492] T 197241807308405baed79aaf07c25fa4 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:36.798429 18097 tablet_bootstrap.cc:654] T 197241807308405baed79aaf07c25fa4 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.798655 18051 ts_tablet_manager.cc:1398] T 5bec006637ea4caebf1ebe64056c7a26 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:36.798844 18051 tablet_bootstrap.cc:492] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:36.799929 18051 tablet_bootstrap.cc:654] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.801996 7000 tablet_service.cc:1467] Processing CreateTablet for tablet cc2d83ce05384dc9b3e306569fa200b3 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=3eb8fcee8b36492886e159a251f6a824]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.802273 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2d83ce05384dc9b3e306569fa200b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.804670 18100 tablet_bootstrap.cc:492] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:36.805459 9350 tablet_service.cc:1467] Processing CreateTablet for tablet cc2d83ce05384dc9b3e306569fa200b3 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=3eb8fcee8b36492886e159a251f6a824]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.805749 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2d83ce05384dc9b3e306569fa200b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.806270 18100 tablet_bootstrap.cc:654] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.807339 18100 tablet_bootstrap.cc:492] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:36.807407 18100 ts_tablet_manager.cc:1367] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:36.808070 18102 tablet_bootstrap.cc:492] T cc2d83ce05384dc9b3e306569fa200b3 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:36.809398 18102 tablet_bootstrap.cc:654] T cc2d83ce05384dc9b3e306569fa200b3 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.810556 18092 tablet_bootstrap.cc:492] T 197241807308405baed79aaf07c25fa4 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:36.811126 18050 tablet_bootstrap.cc:492] T 5bec006637ea4caebf1ebe64056c7a26 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:36.811220 18050 ts_tablet_manager.cc:1367] T 5bec006637ea4caebf1ebe64056c7a26 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 07:59:36.807796 18100 raft_consensus.cc:348] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:36.811664 18100 raft_consensus.cc:374] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.811714 18100 raft_consensus.cc:729] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.811738 18050 raft_consensus.cc:348] T 5bec006637ea4caebf1ebe64056c7a26 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:36.811849 18050 raft_consensus.cc:374] T 5bec006637ea4caebf1ebe64056c7a26 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.811882 18050 raft_consensus.cc:729] T 5bec006637ea4caebf1ebe64056c7a26 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.812003 18050 consensus_queue.cc:260] T 5bec006637ea4caebf1ebe64056c7a26 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:36.812296 18050 ts_tablet_manager.cc:1398] T 5bec006637ea4caebf1ebe64056c7a26 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.812469 18102 tablet_bootstrap.cc:492] T cc2d83ce05384dc9b3e306569fa200b3 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:36.812562 18102 ts_tablet_manager.cc:1367] T cc2d83ce05384dc9b3e306569fa200b3 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:36.813051 18102 raft_consensus.cc:348] T cc2d83ce05384dc9b3e306569fa200b3 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:36.813162 18102 raft_consensus.cc:374] T cc2d83ce05384dc9b3e306569fa200b3 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.813197 18102 raft_consensus.cc:729] T cc2d83ce05384dc9b3e306569fa200b3 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.813303 18102 consensus_queue.cc:260] T cc2d83ce05384dc9b3e306569fa200b3 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:36.813597 18102 ts_tablet_manager.cc:1398] T cc2d83ce05384dc9b3e306569fa200b3 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.814818 18092 tablet_bootstrap.cc:654] T 197241807308405baed79aaf07c25fa4 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.815043 18100 consensus_queue.cc:260] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:36.815510 18100 ts_tablet_manager.cc:1398] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:36.817984 18051 tablet_bootstrap.cc:492] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:36.818082 18051 ts_tablet_manager.cc:1367] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240503 07:59:36.818569 18051 raft_consensus.cc:348] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.818737 18051 raft_consensus.cc:374] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.818764 18051 raft_consensus.cc:729] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.818866 18051 consensus_queue.cc:260] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.819144 18051 ts_tablet_manager.cc:1398] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.821327 18092 tablet_bootstrap.cc:492] T 197241807308405baed79aaf07c25fa4 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:36.821455 18092 ts_tablet_manager.cc:1367] T 197241807308405baed79aaf07c25fa4 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:36.821974 18092 raft_consensus.cc:348] T 197241807308405baed79aaf07c25fa4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.822104 18092 raft_consensus.cc:374] T 197241807308405baed79aaf07c25fa4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.822134 18092 raft_consensus.cc:729] T 197241807308405baed79aaf07c25fa4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.822263 18092 consensus_queue.cc:260] T 197241807308405baed79aaf07c25fa4 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.823211 18097 tablet_bootstrap.cc:492] T 197241807308405baed79aaf07c25fa4 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:36.823325 18097 ts_tablet_manager.cc:1367] T 197241807308405baed79aaf07c25fa4 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 07:59:36.823870 18097 raft_consensus.cc:348] T 197241807308405baed79aaf07c25fa4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.823984 18097 raft_consensus.cc:374] T 197241807308405baed79aaf07c25fa4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.824012 18097 raft_consensus.cc:729] T 197241807308405baed79aaf07c25fa4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.824141 18097 consensus_queue.cc:260] T 197241807308405baed79aaf07c25fa4 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.824445 18097 ts_tablet_manager.cc:1398] T 197241807308405baed79aaf07c25fa4 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.824575 18097 tablet_bootstrap.cc:492] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:36.825495 18097 tablet_bootstrap.cc:654] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.825845 18092 ts_tablet_manager.cc:1398] T 197241807308405baed79aaf07c25fa4 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:36.829934 18097 tablet_bootstrap.cc:492] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:36.830049 18097 ts_tablet_manager.cc:1367] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:36.830585 18097 raft_consensus.cc:348] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:36.830714 18097 raft_consensus.cc:374] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.830742 18097 raft_consensus.cc:729] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.830864 18097 consensus_queue.cc:260] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:36.831400 18097 ts_tablet_manager.cc:1398] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.839943 6744 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:36.846028 9503 raft_consensus.cc:1261] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.846069 7151 raft_consensus.cc:1261] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.846594 17443 consensus_queue.cc:1035] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.846911 17443 consensus_queue.cc:1035] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.851193 17600 raft_consensus.cc:482] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.851310 17600 raft_consensus.cc:504] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.851688 17600 leader_election.cc:288] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:36.855144 6738 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet c49c7188a3ee46b19ab45a469c94f662 (table rebalancer_dynamic_table [id=9f360a15835845a7a34f04b2b22a8e2b]). Expected version 1 got 0
I20240503 07:59:36.855628 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "197241807308405baed79aaf07c25fa4" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:36.855793 7278 raft_consensus.cc:2454] T 197241807308405baed79aaf07c25fa4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:36.856271 9441 leader_election.cc:302] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:36.856442 17599 raft_consensus.cc:2781] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.856509 17599 raft_consensus.cc:482] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.856551 17599 raft_consensus.cc:3037] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.858460 17599 raft_consensus.cc:504] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.858852 17599 leader_election.cc:288] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:36.860271 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "197241807308405baed79aaf07c25fa4" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:36.860404 7278 raft_consensus.cc:3037] T 197241807308405baed79aaf07c25fa4 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.861502 7278 raft_consensus.cc:2454] T 197241807308405baed79aaf07c25fa4 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:36.862421 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "197241807308405baed79aaf07c25fa4" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:36.862537 9634 raft_consensus.cc:2454] T 197241807308405baed79aaf07c25fa4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:36.862731 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "197241807308405baed79aaf07c25fa4" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:36.862828 9636 raft_consensus.cc:3037] T 197241807308405baed79aaf07c25fa4 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.863986 9636 raft_consensus.cc:2454] T 197241807308405baed79aaf07c25fa4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:36.864470 9441 leader_election.cc:302] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:36.865456 17600 raft_consensus.cc:2781] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.865670 17600 raft_consensus.cc:686] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:36.865819 17600 consensus_queue.cc:237] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:36.869103 6742 catalog_manager.cc:5576] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317) does not have the latest schema for tablet c49c7188a3ee46b19ab45a469c94f662 (table rebalancer_dynamic_table [id=9f360a15835845a7a34f04b2b22a8e2b]). Expected version 1 got 0
I20240503 07:59:36.869163 6746 catalog_manager.cc:5484] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:36.872706 17902 tablet.cc:1722] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:36.873047 17902 tablet.cc:1620] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:36.876284 18141 tablet.cc:1722] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:36.880079 6746 catalog_manager.cc:5576] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317) does not have the latest schema for tablet c49c7188a3ee46b19ab45a469c94f662 (table rebalancer_dynamic_table [id=9f360a15835845a7a34f04b2b22a8e2b]). Expected version 1 got 0
I20240503 07:59:36.880296 18141 tablet.cc:1620] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:36.880717 18142 tablet.cc:1722] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:36.881242 18142 tablet.cc:1620] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:36.887256 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=9f360a15835845a7a34f04b2b22a8e2b] alter complete (version 1)
I20240503 07:59:36.919718 17517 raft_consensus.cc:482] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.919878 17517 raft_consensus.cc:504] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:36.920368 17517 leader_election.cc:288] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:36.920897 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2d83ce05384dc9b3e306569fa200b3" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:36.921049 9766 raft_consensus.cc:2454] T cc2d83ce05384dc9b3e306569fa200b3 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:36.921289 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2d83ce05384dc9b3e306569fa200b3" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:36.921360 6956 leader_election.cc:302] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:36.921434 9374 raft_consensus.cc:2454] T cc2d83ce05384dc9b3e306569fa200b3 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:36.921850 17517 raft_consensus.cc:2781] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.921916 17517 raft_consensus.cc:482] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.921957 17517 raft_consensus.cc:3037] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.923163 17517 raft_consensus.cc:504] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:36.923524 17517 leader_election.cc:288] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:36.924108 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2d83ce05384dc9b3e306569fa200b3" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:36.924229 9766 raft_consensus.cc:3037] T cc2d83ce05384dc9b3e306569fa200b3 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.925207 9766 raft_consensus.cc:2454] T cc2d83ce05384dc9b3e306569fa200b3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:36.925400 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2d83ce05384dc9b3e306569fa200b3" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:36.925520 9374 raft_consensus.cc:3037] T cc2d83ce05384dc9b3e306569fa200b3 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.925953 6956 leader_election.cc:302] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:36.926635 9374 raft_consensus.cc:2454] T cc2d83ce05384dc9b3e306569fa200b3 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:36.927419 18143 raft_consensus.cc:2781] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.927544 18143 raft_consensus.cc:686] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:36.927783 18143 consensus_queue.cc:237] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:36.931496 6740 catalog_manager.cc:5484] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:36.998613 6744 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:37.004945 18141 tablet.cc:1722] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:37.005298 18141 tablet.cc:1620] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:37.005381 18142 tablet.cc:1722] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:37.005703 18142 tablet.cc:1620] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:37.008836 17902 tablet.cc:1722] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:37.009203 17902 tablet.cc:1620] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:37.014900 6746 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=9f360a15835845a7a34f04b2b22a8e2b] alter complete (version 2)
I20240503 07:59:37.029439 17876 raft_consensus.cc:482] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.029579 17876 raft_consensus.cc:504] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.030000 17876 leader_election.cc:288] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:37.030589 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bec006637ea4caebf1ebe64056c7a26" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:37.030747 7151 raft_consensus.cc:2454] T 5bec006637ea4caebf1ebe64056c7a26 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:37.030766 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bec006637ea4caebf1ebe64056c7a26" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:37.030900 9504 raft_consensus.cc:2454] T 5bec006637ea4caebf1ebe64056c7a26 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:37.031216 9572 leader_election.cc:302] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2; no voters:
I20240503 07:59:37.031862 17876 raft_consensus.cc:2781] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.031931 17876 raft_consensus.cc:482] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.031965 17876 raft_consensus.cc:3037] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.033185 17876 raft_consensus.cc:504] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.033576 17876 leader_election.cc:288] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:37.034852 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bec006637ea4caebf1ebe64056c7a26" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:37.034987 9504 raft_consensus.cc:3037] T 5bec006637ea4caebf1ebe64056c7a26 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.036168 9504 raft_consensus.cc:2454] T 5bec006637ea4caebf1ebe64056c7a26 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:37.036543 9572 leader_election.cc:302] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:37.036537 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bec006637ea4caebf1ebe64056c7a26" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:37.036654 7151 raft_consensus.cc:3037] T 5bec006637ea4caebf1ebe64056c7a26 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.036718 17297 raft_consensus.cc:2781] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.036792 17297 raft_consensus.cc:686] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:37.036931 17297 consensus_queue.cc:237] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.037652 7151 raft_consensus.cc:2454] T 5bec006637ea4caebf1ebe64056c7a26 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:37.039927 6744 catalog_manager.cc:5484] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:37.069814 7129 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:37.083384 7000 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:37.083736 6867 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:37.083998 7261 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:37.086773 9616 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:37.088889 9350 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:37.093730 9747 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:37.118460 9483 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:37.118500 6744 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:37.127573 18141 tablet.cc:1722] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:37.134449 18142 tablet.cc:1722] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:37.138962 17902 tablet.cc:1722] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:37.140491 6744 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=9f360a15835845a7a34f04b2b22a8e2b] alter complete (version 3)
I20240503 07:59:37.182032 6888 consensus_queue.cc:237] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:37.183530 7151 raft_consensus.cc:1261] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:37.183957 17443 consensus_queue.cc:1035] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:37.184515 9374 raft_consensus.cc:1261] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:37.184929 15993 consensus_queue.cc:1035] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:37.187140 15993 raft_consensus.cc:2932] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:37.188169 9374 raft_consensus.cc:2932] T 5293a26b65484ae5a9e76469e644c5bc P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:37.189239 7150 raft_consensus.cc:2932] T 5293a26b65484ae5a9e76469e644c5bc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:37.190946 6746 catalog_manager.cc:5484] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:37.196064 7000 tablet_service.cc:1514] Processing DeleteTablet for tablet 5293a26b65484ae5a9e76469e644c5bc with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:37.197996 18176 tablet_replica.cc:330] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:37.198233 18176 raft_consensus.cc:2229] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:37.198611 18176 raft_consensus.cc:2258] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.199431 18176 ts_tablet_manager.cc:1875] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:37.201835 18176 ts_tablet_manager.cc:1888] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:37.201920 18176 log.cc:1196] T 5293a26b65484ae5a9e76469e644c5bc P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5293a26b65484ae5a9e76469e644c5bc
I20240503 07:59:37.202473 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 5293a26b65484ae5a9e76469e644c5bc (table rebalance_test_table_1 [id=7240d8c6aeff4067b4b95979e77f21da]) successfully deleted
I20240503 07:59:37.245914 6744 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:37.246096 6744 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:37.248757 6744 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet c49c7188a3ee46b19ab45a469c94f662
I20240503 07:59:37.249377 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet c49c7188a3ee46b19ab45a469c94f662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:37 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:37.249539 9483 tablet_service.cc:1514] Processing DeleteTablet for tablet c49c7188a3ee46b19ab45a469c94f662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:37 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:37.249688 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet c49c7188a3ee46b19ab45a469c94f662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:37 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:37.249812 18198 tablet_replica.cc:330] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:37.249972 18198 raft_consensus.cc:2229] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:37.250128 18198 raft_consensus.cc:2258] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.250317 18200 tablet_replica.cc:330] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:37.250749 18200 raft_consensus.cc:2229] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:37.250970 18198 ts_tablet_manager.cc:1875] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:37.251041 18200 raft_consensus.cc:2258] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.252094 18200 ts_tablet_manager.cc:1875] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:37.253707 18198 ts_tablet_manager.cc:1888] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:37.253798 18198 log.cc:1196] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c49c7188a3ee46b19ab45a469c94f662
I20240503 07:59:37.254176 18198 ts_tablet_manager.cc:1909] T c49c7188a3ee46b19ab45a469c94f662 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:37.255363 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet c49c7188a3ee46b19ab45a469c94f662 (table rebalancer_dynamic_table_new_name [id=9f360a15835845a7a34f04b2b22a8e2b]) successfully deleted
I20240503 07:59:37.255725 18199 tablet_replica.cc:330] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:37.255950 18199 raft_consensus.cc:2229] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:37.256139 18199 raft_consensus.cc:2258] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.256253 18200 ts_tablet_manager.cc:1888] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:37.256626 18200 log.cc:1196] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c49c7188a3ee46b19ab45a469c94f662
I20240503 07:59:37.256971 18199 ts_tablet_manager.cc:1875] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:37.257346 18200 ts_tablet_manager.cc:1909] T c49c7188a3ee46b19ab45a469c94f662 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:37.258180 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet c49c7188a3ee46b19ab45a469c94f662 (table rebalancer_dynamic_table_new_name [id=9f360a15835845a7a34f04b2b22a8e2b]) successfully deleted
I20240503 07:59:37.259516 18199 ts_tablet_manager.cc:1888] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:37.259994 18199 log.cc:1196] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/c49c7188a3ee46b19ab45a469c94f662
I20240503 07:59:37.260411 18199 ts_tablet_manager.cc:1909] T c49c7188a3ee46b19ab45a469c94f662 P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:37.261173 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet c49c7188a3ee46b19ab45a469c94f662 (table rebalancer_dynamic_table_new_name [id=9f360a15835845a7a34f04b2b22a8e2b]) successfully deleted
I20240503 07:59:37.308053 17599 consensus_queue.cc:1035] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.311676 18139 consensus_queue.cc:1035] T 197241807308405baed79aaf07c25fa4 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.340164 7261 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:37.345525 7000 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:37.348176 9350 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:37.352968 7129 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:37.353489 6745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:37.354300 6867 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:37.363389 9350 tablet_service.cc:1467] Processing CreateTablet for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b18e478f2f604d76b1dcd9e5af2d66a2]), partition=
I20240503 07:59:37.363683 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.367902 9747 tablet_service.cc:1467] Processing CreateTablet for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b18e478f2f604d76b1dcd9e5af2d66a2]), partition=
I20240503 07:59:37.368160 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.370491 18238 tablet_bootstrap.cc:492] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:37.370504 9616 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:37.371397 7129 tablet_service.cc:1467] Processing CreateTablet for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b18e478f2f604d76b1dcd9e5af2d66a2]), partition=
I20240503 07:59:37.371533 18238 tablet_bootstrap.cc:654] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.371631 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.382519 9483 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:37.396363 9747 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:37.407750 17517 consensus_queue.cc:1035] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.419170 18238 tablet_bootstrap.cc:492] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:37.419320 18238 ts_tablet_manager.cc:1367] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.000s
I20240503 07:59:37.419798 18238 raft_consensus.cc:348] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.419896 18238 raft_consensus.cc:374] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.419916 18238 raft_consensus.cc:729] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.420032 18238 consensus_queue.cc:260] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.420311 18238 ts_tablet_manager.cc:1398] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.425796 18143 consensus_queue.cc:1035] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.428825 18237 tablet_bootstrap.cc:492] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:37.429251 18240 tablet_bootstrap.cc:492] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:37.430701 18240 tablet_bootstrap.cc:654] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.434746 18237 tablet_bootstrap.cc:654] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.439952 18240 tablet_bootstrap.cc:492] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:37.440538 18240 ts_tablet_manager.cc:1367] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240503 07:59:37.441179 18240 raft_consensus.cc:348] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.441313 18240 raft_consensus.cc:374] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.441349 18240 raft_consensus.cc:729] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.441473 18240 consensus_queue.cc:260] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.442581 18240 ts_tablet_manager.cc:1398] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:37.443461 18237 tablet_bootstrap.cc:492] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:37.443847 18237 ts_tablet_manager.cc:1367] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:37.447703 18237 raft_consensus.cc:348] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.448001 18237 raft_consensus.cc:374] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.448056 18237 raft_consensus.cc:729] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.448195 18237 consensus_queue.cc:260] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.448498 18237 ts_tablet_manager.cc:1398] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.485695 9374 consensus_queue.cc:237] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.486953 7151 raft_consensus.cc:1261] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:37.487129 9765 raft_consensus.cc:1261] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:37.487484 18080 consensus_queue.cc:1035] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:37.487946 17499 consensus_queue.cc:1035] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:37.490494 18080 raft_consensus.cc:2932] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:37.490926 9766 raft_consensus.cc:2932] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:37.491003 7151 raft_consensus.cc:2932] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:37.494309 6745 catalog_manager.cc:5484] T 866c1bf17d444315bb3eb83a7a283b54 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index 2 to 3, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:37.497962 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet 866c1bf17d444315bb3eb83a7a283b54 with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:37.498468 18273 tablet_replica.cc:330] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:37.498667 18273 raft_consensus.cc:2229] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:37.499490 18273 raft_consensus.cc:2258] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.500068 18273 ts_tablet_manager.cc:1875] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:37.502925 18273 ts_tablet_manager.cc:1888] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:37.503010 18273 log.cc:1196] T 866c1bf17d444315bb3eb83a7a283b54 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/866c1bf17d444315bb3eb83a7a283b54
I20240503 07:59:37.503644 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 866c1bf17d444315bb3eb83a7a283b54 (table rebalancer_extra_table_4 [id=9b4298d7d24e485d91d7ae96bac125e2]) successfully deleted
I20240503 07:59:37.509599 17297 consensus_queue.cc:1035] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.512732 17297 consensus_queue.cc:1035] T 5bec006637ea4caebf1ebe64056c7a26 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.530771 17659 raft_consensus.cc:482] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.530892 17659 raft_consensus.cc:504] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.531371 17659 leader_election.cc:288] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:37.531745 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2ca3d01e24a1bb29b5ab724b7eaa7" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:37.531896 7151 raft_consensus.cc:2454] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:37.531745 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2ca3d01e24a1bb29b5ab724b7eaa7" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:37.532073 9374 raft_consensus.cc:2454] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:37.532348 9703 leader_election.cc:302] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:37.532521 17659 raft_consensus.cc:2781] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.532579 17659 raft_consensus.cc:482] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.532622 17659 raft_consensus.cc:3037] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.534449 17659 raft_consensus.cc:504] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.535142 17659 leader_election.cc:288] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:37.535166 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2ca3d01e24a1bb29b5ab724b7eaa7" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:37.535161 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2ca3d01e24a1bb29b5ab724b7eaa7" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:37.535292 9374 raft_consensus.cc:3037] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.535295 7151 raft_consensus.cc:3037] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.536232 7151 raft_consensus.cc:2454] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:37.536396 9374 raft_consensus.cc:2454] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:37.536492 9703 leader_election.cc:302] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:37.537040 17659 raft_consensus.cc:2781] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.537142 17659 raft_consensus.cc:686] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:37.537297 17659 consensus_queue.cc:237] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.539889 6740 catalog_manager.cc:5484] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:37.559163 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:37.568046 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 84811c45da9a4088a6acf5369a728caf (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2a4d988c09504687a91b0578ecb1b6ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.568166 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 84811c45da9a4088a6acf5369a728caf (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2a4d988c09504687a91b0578ecb1b6ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.568409 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84811c45da9a4088a6acf5369a728caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.568475 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84811c45da9a4088a6acf5369a728caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.568885 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 2c87f57fdf134ec7a2f6cba79d400b8a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2a4d988c09504687a91b0578ecb1b6ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.569149 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c87f57fdf134ec7a2f6cba79d400b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.571086 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 5da90bdd093d4fe1881443920b8aa964 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2a4d988c09504687a91b0578ecb1b6ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.571313 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da90bdd093d4fe1881443920b8aa964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.571585 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 5da90bdd093d4fe1881443920b8aa964 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2a4d988c09504687a91b0578ecb1b6ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.571799 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da90bdd093d4fe1881443920b8aa964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.573477 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 2c87f57fdf134ec7a2f6cba79d400b8a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2a4d988c09504687a91b0578ecb1b6ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.573750 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c87f57fdf134ec7a2f6cba79d400b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.573985 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 84811c45da9a4088a6acf5369a728caf (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2a4d988c09504687a91b0578ecb1b6ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.574270 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84811c45da9a4088a6acf5369a728caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.576661 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 5da90bdd093d4fe1881443920b8aa964 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2a4d988c09504687a91b0578ecb1b6ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.576939 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da90bdd093d4fe1881443920b8aa964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.577872 18315 tablet_bootstrap.cc:492] T 2c87f57fdf134ec7a2f6cba79d400b8a P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:37.577967 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 2c87f57fdf134ec7a2f6cba79d400b8a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=2a4d988c09504687a91b0578ecb1b6ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.578253 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c87f57fdf134ec7a2f6cba79d400b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.578909 18315 tablet_bootstrap.cc:654] T 2c87f57fdf134ec7a2f6cba79d400b8a P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.580657 18238 tablet_bootstrap.cc:492] T 84811c45da9a4088a6acf5369a728caf P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:37.580829 18240 tablet_bootstrap.cc:492] T 2c87f57fdf134ec7a2f6cba79d400b8a P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:37.581642 18238 tablet_bootstrap.cc:654] T 84811c45da9a4088a6acf5369a728caf P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.581671 18240 tablet_bootstrap.cc:654] T 2c87f57fdf134ec7a2f6cba79d400b8a P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.583655 18314 tablet_bootstrap.cc:492] T 84811c45da9a4088a6acf5369a728caf P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:37.584708 18314 tablet_bootstrap.cc:654] T 84811c45da9a4088a6acf5369a728caf P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.588771 18317 tablet_bootstrap.cc:492] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:37.589033 18316 tablet_bootstrap.cc:492] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:37.589951 18238 tablet_bootstrap.cc:492] T 84811c45da9a4088a6acf5369a728caf P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:37.590063 18238 ts_tablet_manager.cc:1367] T 84811c45da9a4088a6acf5369a728caf P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:37.590113 18316 tablet_bootstrap.cc:654] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.590610 18238 raft_consensus.cc:348] T 84811c45da9a4088a6acf5369a728caf P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.590744 18238 raft_consensus.cc:374] T 84811c45da9a4088a6acf5369a728caf P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.590773 18238 raft_consensus.cc:729] T 84811c45da9a4088a6acf5369a728caf P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.590915 18238 consensus_queue.cc:260] T 84811c45da9a4088a6acf5369a728caf P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.591219 18238 ts_tablet_manager.cc:1398] T 84811c45da9a4088a6acf5369a728caf P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.591662 18240 tablet_bootstrap.cc:492] T 2c87f57fdf134ec7a2f6cba79d400b8a P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:37.591764 18240 ts_tablet_manager.cc:1367] T 2c87f57fdf134ec7a2f6cba79d400b8a P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240503 07:59:37.592196 18313 tablet_bootstrap.cc:492] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:37.592415 18240 raft_consensus.cc:348] T 2c87f57fdf134ec7a2f6cba79d400b8a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.592541 18240 raft_consensus.cc:374] T 2c87f57fdf134ec7a2f6cba79d400b8a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.592578 18240 raft_consensus.cc:729] T 2c87f57fdf134ec7a2f6cba79d400b8a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.592695 18240 consensus_queue.cc:260] T 2c87f57fdf134ec7a2f6cba79d400b8a P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.592965 18240 ts_tablet_manager.cc:1398] T 2c87f57fdf134ec7a2f6cba79d400b8a P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:37.597427 18317 tablet_bootstrap.cc:654] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.597895 18313 tablet_bootstrap.cc:654] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.603610 18317 tablet_bootstrap.cc:492] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:37.603796 18317 ts_tablet_manager.cc:1367] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240503 07:59:37.604333 18315 tablet_bootstrap.cc:492] T 2c87f57fdf134ec7a2f6cba79d400b8a P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:37.604444 18315 ts_tablet_manager.cc:1367] T 2c87f57fdf134ec7a2f6cba79d400b8a P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.002s
I20240503 07:59:37.604964 18315 raft_consensus.cc:348] T 2c87f57fdf134ec7a2f6cba79d400b8a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.605185 18315 raft_consensus.cc:374] T 2c87f57fdf134ec7a2f6cba79d400b8a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.605233 18315 raft_consensus.cc:729] T 2c87f57fdf134ec7a2f6cba79d400b8a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.605751 18317 raft_consensus.cc:348] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:37.605927 18313 tablet_bootstrap.cc:492] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:37.605934 18317 raft_consensus.cc:374] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.605979 18317 raft_consensus.cc:729] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.606098 18317 consensus_queue.cc:260] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:37.606166 18313 ts_tablet_manager.cc:1367] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:37.607419 18317 ts_tablet_manager.cc:1398] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:37.607357 18313 raft_consensus.cc:348] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.607474 18313 raft_consensus.cc:374] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.607566 18313 raft_consensus.cc:729] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.607774 18313 consensus_queue.cc:260] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.608160 18313 ts_tablet_manager.cc:1398] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:37.608425 18313 tablet_bootstrap.cc:492] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:37.611888 18313 tablet_bootstrap.cc:654] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.612704 18315 consensus_queue.cc:260] T 2c87f57fdf134ec7a2f6cba79d400b8a P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.614066 18313 tablet_bootstrap.cc:492] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:37.614276 18313 ts_tablet_manager.cc:1367] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:37.614862 18313 raft_consensus.cc:348] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:37.615720 18313 raft_consensus.cc:374] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.615964 18315 ts_tablet_manager.cc:1398] T 2c87f57fdf134ec7a2f6cba79d400b8a P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 07:59:37.616091 18313 raft_consensus.cc:729] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.616250 18313 consensus_queue.cc:260] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:37.617300 18313 ts_tablet_manager.cc:1398] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:37.619562 18316 tablet_bootstrap.cc:492] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:37.619712 18316 ts_tablet_manager.cc:1367] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20240503 07:59:37.620416 18316 raft_consensus.cc:348] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.620528 18316 raft_consensus.cc:374] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.620556 18316 raft_consensus.cc:729] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.620702 18316 consensus_queue.cc:260] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.621076 18316 ts_tablet_manager.cc:1398] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.627435 18314 tablet_bootstrap.cc:492] T 84811c45da9a4088a6acf5369a728caf P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:37.627660 18314 ts_tablet_manager.cc:1367] T 84811c45da9a4088a6acf5369a728caf P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20240503 07:59:37.629072 18314 raft_consensus.cc:348] T 84811c45da9a4088a6acf5369a728caf P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.629288 18314 raft_consensus.cc:374] T 84811c45da9a4088a6acf5369a728caf P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.629361 18314 raft_consensus.cc:729] T 84811c45da9a4088a6acf5369a728caf P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.629549 18314 consensus_queue.cc:260] T 84811c45da9a4088a6acf5369a728caf P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.630796 18236 raft_consensus.cc:482] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.630903 18236 raft_consensus.cc:504] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.631356 18236 leader_election.cc:288] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:37.632160 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c87f57fdf134ec7a2f6cba79d400b8a" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:37.632340 7020 raft_consensus.cc:2454] T 2c87f57fdf134ec7a2f6cba79d400b8a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:37.632622 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c87f57fdf134ec7a2f6cba79d400b8a" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:37.632813 7151 raft_consensus.cc:2454] T 2c87f57fdf134ec7a2f6cba79d400b8a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 0.
I20240503 07:59:37.633610 7218 leader_election.cc:302] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:37.634446 17989 raft_consensus.cc:2781] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.634517 17989 raft_consensus.cc:482] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.634572 17989 raft_consensus.cc:3037] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.635967 17989 raft_consensus.cc:504] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.636377 17989 leader_election.cc:288] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:37.637096 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c87f57fdf134ec7a2f6cba79d400b8a" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:37.637246 7151 raft_consensus.cc:3037] T 2c87f57fdf134ec7a2f6cba79d400b8a P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.637739 18314 ts_tablet_manager.cc:1398] T 84811c45da9a4088a6acf5369a728caf P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240503 07:59:37.638437 7151 raft_consensus.cc:2454] T 2c87f57fdf134ec7a2f6cba79d400b8a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:37.639068 18314 tablet_bootstrap.cc:492] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:37.639250 7218 leader_election.cc:302] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:37.639727 17989 raft_consensus.cc:2781] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.639747 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c87f57fdf134ec7a2f6cba79d400b8a" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:37.639815 17989 raft_consensus.cc:686] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:37.639869 7020 raft_consensus.cc:3037] T 2c87f57fdf134ec7a2f6cba79d400b8a P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.639952 17989 consensus_queue.cc:237] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:37.641039 7020 raft_consensus.cc:2454] T 2c87f57fdf134ec7a2f6cba79d400b8a P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:37.643127 6740 catalog_manager.cc:5484] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:37.647420 18314 tablet_bootstrap.cc:654] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.648769 18314 tablet_bootstrap.cc:492] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:37.648890 18314 ts_tablet_manager.cc:1367] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:37.649448 18314 raft_consensus.cc:348] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:37.649673 18314 raft_consensus.cc:374] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.649763 18314 raft_consensus.cc:729] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.649969 18314 consensus_queue.cc:260] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:37.651451 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:37.664748 7129 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:37.670181 9350 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:37.676466 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:37.685123 18314 ts_tablet_manager.cc:1398] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240503 07:59:37.687969 7151 raft_consensus.cc:1261] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.687987 9374 raft_consensus.cc:1261] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.688462 17401 consensus_queue.cc:1035] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.688756 17401 consensus_queue.cc:1035] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.693526 6746 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 (table rebalancer_dynamic_table [id=b18e478f2f604d76b1dcd9e5af2d66a2]). Expected version 1 got 0
I20240503 07:59:37.693962 6740 catalog_manager.cc:5576] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317) does not have the latest schema for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 (table rebalancer_dynamic_table [id=b18e478f2f604d76b1dcd9e5af2d66a2]). Expected version 1 got 0
I20240503 07:59:37.698118 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:37.701383 18341 tablet.cc:1722] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:37.702817 18341 tablet.cc:1620] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:37.710806 6738 catalog_manager.cc:5576] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815) does not have the latest schema for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 (table rebalancer_dynamic_table [id=b18e478f2f604d76b1dcd9e5af2d66a2]). Expected version 1 got 0
I20240503 07:59:37.713395 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=b18e478f2f604d76b1dcd9e5af2d66a2] alter complete (version 1)
I20240503 07:59:37.713379 18342 tablet.cc:1722] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:37.713727 18342 tablet.cc:1620] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:37.714809 7000 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:37.716425 9615 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:37.720366 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:37.728201 6740 catalog_manager.cc:5576] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317) does not have the latest schema for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 (table rebalancer_dynamic_table [id=b18e478f2f604d76b1dcd9e5af2d66a2]). Expected version 1 got 0
I20240503 07:59:37.742820 18343 tablet.cc:1722] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:37.743245 18343 tablet.cc:1620] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:37.743847 9483 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:37.805037 7151 consensus_queue.cc:237] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:37.806505 7282 raft_consensus.cc:1261] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:37.807188 17804 consensus_queue.cc:1035] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:37.807475 6886 raft_consensus.cc:1261] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:37.808391 17804 consensus_queue.cc:1035] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:37.809527 17340 raft_consensus.cc:2932] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:37.809931 7282 raft_consensus.cc:2932] T 79dd3d8c050d4bba829de57fe39b025f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:37.810592 6886 raft_consensus.cc:2932] T 79dd3d8c050d4bba829de57fe39b025f P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:37.813838 6740 catalog_manager.cc:5484] T 79dd3d8c050d4bba829de57fe39b025f P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:37.817479 9483 tablet_service.cc:1514] Processing DeleteTablet for tablet 79dd3d8c050d4bba829de57fe39b025f with delete_type TABLET_DATA_TOMBSTONED (TS cb73994515af4880b05383e78368bb4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:37.818656 18362 tablet_replica.cc:330] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:37.818845 18362 raft_consensus.cc:2229] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:37.818989 18362 raft_consensus.cc:2258] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.819576 18362 ts_tablet_manager.cc:1875] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:37.821171 17876 raft_consensus.cc:482] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.821269 17876 raft_consensus.cc:504] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:37.821688 17876 leader_election.cc:288] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:37.821728 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:37.822305 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da90bdd093d4fe1881443920b8aa964" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:37.822464 9504 raft_consensus.cc:2454] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:37.823032 9572 leader_election.cc:302] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:37.823103 17600 raft_consensus.cc:482] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.823201 17600 raft_consensus.cc:504] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:37.823431 17876 raft_consensus.cc:2781] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.823477 17876 raft_consensus.cc:482] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.823500 17876 raft_consensus.cc:3037] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.823549 17600 leader_election.cc:288] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:37.823983 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da90bdd093d4fe1881443920b8aa964" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:37.824334 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da90bdd093d4fe1881443920b8aa964" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:37.824468 6888 raft_consensus.cc:2454] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:37.824548 18362 ts_tablet_manager.cc:1888] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:37.824602 18362 log.cc:1196] T 79dd3d8c050d4bba829de57fe39b025f P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/79dd3d8c050d4bba829de57fe39b025f
I20240503 07:59:37.824652 17876 raft_consensus.cc:504] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:37.824913 9438 leader_election.cc:302] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:37.825052 17876 leader_election.cc:288] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:37.825462 17600 raft_consensus.cc:2781] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.825536 17600 raft_consensus.cc:482] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.825507 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da90bdd093d4fe1881443920b8aa964" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:37.825575 17600 raft_consensus.cc:3037] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.825605 6888 raft_consensus.cc:3037] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.825616 9634 raft_consensus.cc:2379] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb73994515af4880b05383e78368bb4a in current term 1: Already voted for candidate 159177b3caeb434baa98e04136d050b2 in this term.
I20240503 07:59:37.825716 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da90bdd093d4fe1881443920b8aa964" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:37.826102 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 79dd3d8c050d4bba829de57fe39b025f (table rebalancer_extra_table_20 [id=2f64f0c37c8a48f68cd6e052fc7420fb]) successfully deleted
I20240503 07:59:37.826555 6888 raft_consensus.cc:2454] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:37.826560 17600 raft_consensus.cc:504] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:37.826552 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da90bdd093d4fe1881443920b8aa964" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:37.826714 6886 raft_consensus.cc:2362] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1. Re-sending same reply.
I20240503 07:59:37.826923 17600 leader_election.cc:288] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:37.826912 9569 leader_election.cc:302] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:37.827152 9504 raft_consensus.cc:2379] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 159177b3caeb434baa98e04136d050b2 in current term 1: Already voted for candidate cb73994515af4880b05383e78368bb4a in this term.
I20240503 07:59:37.827292 17876 raft_consensus.cc:2781] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.827378 17876 raft_consensus.cc:686] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:37.827502 17876 consensus_queue.cc:237] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:37.830286 6747 catalog_manager.cc:5484] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:37.833732 18341 tablet.cc:1722] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:37.834123 18341 tablet.cc:1620] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:37.834264 18343 tablet.cc:1722] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:37.834630 18343 tablet.cc:1620] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: MemRowSet was empty: no flush needed.
I20240503 07:59:37.837430 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da90bdd093d4fe1881443920b8aa964" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:37.838642 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da90bdd093d4fe1881443920b8aa964" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:37.838838 6886 raft_consensus.cc:2379] T 5da90bdd093d4fe1881443920b8aa964 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cb73994515af4880b05383e78368bb4a in current term 1: Already voted for candidate 159177b3caeb434baa98e04136d050b2 in this term.
I20240503 07:59:37.840296 9438 leader_election.cc:302] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb73994515af4880b05383e78368bb4a; no voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:37.841408 17600 raft_consensus.cc:2726] T 5da90bdd093d4fe1881443920b8aa964 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 07:59:37.841886 18342 tablet.cc:1722] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:37.842229 18342 tablet.cc:1620] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:37.843763 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=b18e478f2f604d76b1dcd9e5af2d66a2] alter complete (version 2)
I20240503 07:59:37.867286 17297 raft_consensus.cc:482] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.867431 17297 raft_consensus.cc:504] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.867872 17297 leader_election.cc:288] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:37.868227 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84811c45da9a4088a6acf5369a728caf" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:37.868376 6886 raft_consensus.cc:2454] T 84811c45da9a4088a6acf5369a728caf P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:37.868628 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84811c45da9a4088a6acf5369a728caf" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:37.868675 9569 leader_election.cc:302] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:37.868764 9766 raft_consensus.cc:2454] T 84811c45da9a4088a6acf5369a728caf P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:37.869038 17876 raft_consensus.cc:2781] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.869102 17876 raft_consensus.cc:482] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.869128 17876 raft_consensus.cc:3037] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.870127 17876 raft_consensus.cc:504] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.870497 17876 leader_election.cc:288] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:37.870779 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84811c45da9a4088a6acf5369a728caf" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:37.870906 9766 raft_consensus.cc:3037] T 84811c45da9a4088a6acf5369a728caf P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.872092 9766 raft_consensus.cc:2454] T 84811c45da9a4088a6acf5369a728caf P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:37.872320 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84811c45da9a4088a6acf5369a728caf" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:37.872442 6888 raft_consensus.cc:3037] T 84811c45da9a4088a6acf5369a728caf P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.873225 6888 raft_consensus.cc:2454] T 84811c45da9a4088a6acf5369a728caf P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:37.873550 9569 leader_election.cc:302] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:37.873725 17297 raft_consensus.cc:2781] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.873840 17297 raft_consensus.cc:686] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:37.873975 17297 consensus_queue.cc:237] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:37.876935 6738 catalog_manager.cc:5484] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:37.946975 6744 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:37.956383 18341 tablet.cc:1722] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:37.956857 18343 tablet.cc:1722] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:37.962652 6744 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=b18e478f2f604d76b1dcd9e5af2d66a2] alter complete (version 3)
I20240503 07:59:37.970674 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:37.971434 18342 tablet.cc:1722] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:37.983601 9615 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:37.985755 9350 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:37.992338 7129 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:37.993584 7000 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:37.998785 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:38.012387 9483 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:38.037415 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:38.073246 6739 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:38.073441 6739 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:38.076818 6739 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7
I20240503 07:59:38.077432 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:38 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:38.077735 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:38 UTC) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:38.078123 18419 tablet_replica.cc:330] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:38.078189 18420 tablet_replica.cc:330] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:38.078380 18420 raft_consensus.cc:2229] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.078380 18419 raft_consensus.cc:2229] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.078478 9747 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:38 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:38.078521 18419 raft_consensus.cc:2258] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.078521 18420 raft_consensus.cc:2258] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.078866 18421 tablet_replica.cc:330] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:38.079082 18421 raft_consensus.cc:2229] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:38.079377 18421 raft_consensus.cc:2258] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.079424 18419 ts_tablet_manager.cc:1875] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:38.080801 18421 ts_tablet_manager.cc:1875] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:38.082082 18419 ts_tablet_manager.cc:1888] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:38.082638 18420 ts_tablet_manager.cc:1875] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:38.083467 18421 ts_tablet_manager.cc:1888] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:38.083739 18421 log.cc:1196] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/e9e2ca3d01e24a1bb29b5ab724b7eaa7
I20240503 07:59:38.083811 18419 log.cc:1196] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/e9e2ca3d01e24a1bb29b5ab724b7eaa7
I20240503 07:59:38.084035 18421 ts_tablet_manager.cc:1909] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:38.084069 18419 ts_tablet_manager.cc:1909] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:38.084805 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 (table rebalancer_dynamic_table_new_name [id=b18e478f2f604d76b1dcd9e5af2d66a2]) successfully deleted
I20240503 07:59:38.084825 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 (table rebalancer_dynamic_table_new_name [id=b18e478f2f604d76b1dcd9e5af2d66a2]) successfully deleted
I20240503 07:59:38.086251 18420 ts_tablet_manager.cc:1888] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:38.086393 18420 log.cc:1196] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e9e2ca3d01e24a1bb29b5ab724b7eaa7
I20240503 07:59:38.086709 18420 ts_tablet_manager.cc:1909] T e9e2ca3d01e24a1bb29b5ab724b7eaa7 P 00a3f8e159c64327832d79fb32cdcca8: Deleting consensus metadata
I20240503 07:59:38.087406 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet e9e2ca3d01e24a1bb29b5ab724b7eaa7 (table rebalancer_dynamic_table_new_name [id=b18e478f2f604d76b1dcd9e5af2d66a2]) successfully deleted
I20240503 07:59:38.116240 9374 consensus_queue.cc:237] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:38.117623 6888 raft_consensus.cc:1261] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:38.117687 9636 raft_consensus.cc:1261] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:38.118146 17499 consensus_queue.cc:1035] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:38.118379 18264 consensus_queue.cc:1035] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:38.120245 17499 raft_consensus.cc:2932] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:38.120721 6886 raft_consensus.cc:2932] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:38.120805 9636 raft_consensus.cc:2932] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:38.124109 6739 catalog_manager.cc:5484] T dc3367cc41234d52b74eee94d6e67069 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:38.127595 9747 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3367cc41234d52b74eee94d6e67069 with delete_type TABLET_DATA_TOMBSTONED (TS a372e98a670f49e9adca46ab5c836c4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:38.127910 18421 tablet_replica.cc:330] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:38.128079 18421 raft_consensus.cc:2229] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.128225 18421 raft_consensus.cc:2258] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.128798 18421 ts_tablet_manager.cc:1875] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:38.131145 18421 ts_tablet_manager.cc:1888] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:38.131230 18421 log.cc:1196] T dc3367cc41234d52b74eee94d6e67069 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/dc3367cc41234d52b74eee94d6e67069
I20240503 07:59:38.131736 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet dc3367cc41234d52b74eee94d6e67069 (table rebalancer_extra_table_34 [id=eba63fcb11c54ebf8beaa0fe0e7a772a]) successfully deleted
I20240503 07:59:38.158396 18236 consensus_queue.cc:1035] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.161948 17989 consensus_queue.cc:1035] T 2c87f57fdf134ec7a2f6cba79d400b8a P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.183549 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:38.190029 9483 tablet_service.cc:1467] Processing CreateTablet for tablet 1480785341e049a3b163c3bc97b4736b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2334ae647b314f1bb0417bec0dc5b86a]), partition=
I20240503 07:59:38.190379 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1480785341e049a3b163c3bc97b4736b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.191350 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 1480785341e049a3b163c3bc97b4736b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2334ae647b314f1bb0417bec0dc5b86a]), partition=
I20240503 07:59:38.191602 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1480785341e049a3b163c3bc97b4736b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.193953 18486 tablet_bootstrap.cc:492] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:38.194769 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 1480785341e049a3b163c3bc97b4736b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2334ae647b314f1bb0417bec0dc5b86a]), partition=
I20240503 07:59:38.195022 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1480785341e049a3b163c3bc97b4736b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.195140 18486 tablet_bootstrap.cc:654] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.197224 18486 tablet_bootstrap.cc:492] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:38.197324 18486 ts_tablet_manager.cc:1367] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:38.197515 18488 tablet_bootstrap.cc:492] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:38.197810 18486 raft_consensus.cc:348] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:38.197942 18486 raft_consensus.cc:374] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.197981 18486 raft_consensus.cc:729] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.198123 18486 consensus_queue.cc:260] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:38.198467 18486 ts_tablet_manager.cc:1398] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:38.198585 18488 tablet_bootstrap.cc:654] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.199718 18488 tablet_bootstrap.cc:492] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:38.199812 18488 ts_tablet_manager.cc:1367] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:38.200294 18488 raft_consensus.cc:348] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:38.200405 18488 raft_consensus.cc:374] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.200438 18488 raft_consensus.cc:729] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.200556 18488 consensus_queue.cc:260] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:38.200855 18488 ts_tablet_manager.cc:1398] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.206588 18485 tablet_bootstrap.cc:492] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:38.210721 18485 tablet_bootstrap.cc:654] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.212002 18485 tablet_bootstrap.cc:492] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:38.212116 18485 ts_tablet_manager.cc:1367] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:38.212654 18485 raft_consensus.cc:348] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:38.212807 18485 raft_consensus.cc:374] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.212838 18485 raft_consensus.cc:729] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.212996 18485 consensus_queue.cc:260] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:38.213689 18485 ts_tablet_manager.cc:1398] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:38.239220 17297 consensus_queue.cc:1035] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.243953 17876 consensus_queue.cc:1035] T 5da90bdd093d4fe1881443920b8aa964 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.251928 17876 consensus_queue.cc:1035] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.258134 17297 consensus_queue.cc:1035] T 84811c45da9a4088a6acf5369a728caf P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.277007 17499 raft_consensus.cc:482] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.277201 17499 raft_consensus.cc:504] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:38.277678 17499 leader_election.cc:288] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:38.278419 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1480785341e049a3b163c3bc97b4736b" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:38.278584 7282 raft_consensus.cc:2454] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:38.278972 9310 leader_election.cc:302] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:38.279038 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1480785341e049a3b163c3bc97b4736b" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:38.279173 9504 raft_consensus.cc:2454] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:38.279507 17499 raft_consensus.cc:2781] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.279577 17499 raft_consensus.cc:482] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.279616 17499 raft_consensus.cc:3037] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.280808 17499 raft_consensus.cc:504] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:38.281219 17499 leader_election.cc:288] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:38.282125 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1480785341e049a3b163c3bc97b4736b" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:38.282256 9504 raft_consensus.cc:3037] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.282317 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1480785341e049a3b163c3bc97b4736b" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:38.282480 7282 raft_consensus.cc:3037] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.283196 9504 raft_consensus.cc:2454] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:38.283610 7282 raft_consensus.cc:2454] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:38.283950 9310 leader_election.cc:302] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:38.284133 17499 raft_consensus.cc:2781] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.284231 17499 raft_consensus.cc:686] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:38.284379 17499 consensus_queue.cc:237] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:38.293671 6745 catalog_manager.cc:5484] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:38.312813 9350 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:38.322949 7129 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:38.324096 9615 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:38.326649 7000 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:38.331367 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:38.339054 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:38.339852 9747 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:38.344563 9483 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:38.415025 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:38.416427 6745 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:38.423316 9504 raft_consensus.cc:1261] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.424456 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 801d28d7a82549b1ad71fbc5e939c261 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=555ad53a81194324b1dd6c1130fe3e6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.424738 7282 raft_consensus.cc:1261] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.424851 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801d28d7a82549b1ad71fbc5e939c261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.425719 17499 consensus_queue.cc:1035] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.426154 18264 consensus_queue.cc:1035] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.426431 9745 tablet_service.cc:1467] Processing CreateTablet for tablet 664327fac0a24b95aa459d16efe2ef43 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=555ad53a81194324b1dd6c1130fe3e6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.426724 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 664327fac0a24b95aa459d16efe2ef43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.427816 18526 tablet_bootstrap.cc:492] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:38.427779 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 801d28d7a82549b1ad71fbc5e939c261 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=555ad53a81194324b1dd6c1130fe3e6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.428054 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801d28d7a82549b1ad71fbc5e939c261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.429466 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 664327fac0a24b95aa459d16efe2ef43 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=555ad53a81194324b1dd6c1130fe3e6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.429728 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 664327fac0a24b95aa459d16efe2ef43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.430617 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa65337d4dc48df91409554d00f5f17 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=555ad53a81194324b1dd6c1130fe3e6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.430876 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa65337d4dc48df91409554d00f5f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.431232 18526 tablet_bootstrap.cc:654] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.432005 18527 tablet_bootstrap.cc:492] T 664327fac0a24b95aa459d16efe2ef43 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:38.432497 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa65337d4dc48df91409554d00f5f17 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=555ad53a81194324b1dd6c1130fe3e6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.432763 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa65337d4dc48df91409554d00f5f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.433445 18488 tablet_bootstrap.cc:492] T 801d28d7a82549b1ad71fbc5e939c261 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:38.434479 18488 tablet_bootstrap.cc:654] T 801d28d7a82549b1ad71fbc5e939c261 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.435073 18527 tablet_bootstrap.cc:654] T 664327fac0a24b95aa459d16efe2ef43 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.435855 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 664327fac0a24b95aa459d16efe2ef43 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=555ad53a81194324b1dd6c1130fe3e6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.436148 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 664327fac0a24b95aa459d16efe2ef43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.426431 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 801d28d7a82549b1ad71fbc5e939c261 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=555ad53a81194324b1dd6c1130fe3e6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.436719 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801d28d7a82549b1ad71fbc5e939c261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.438791 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa65337d4dc48df91409554d00f5f17 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=555ad53a81194324b1dd6c1130fe3e6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.439080 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa65337d4dc48df91409554d00f5f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.439829 18526 tablet_bootstrap.cc:492] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:38.439935 18526 ts_tablet_manager.cc:1367] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:38.440618 18526 raft_consensus.cc:348] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.440757 18526 raft_consensus.cc:374] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.440786 18526 raft_consensus.cc:729] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.440982 18526 consensus_queue.cc:260] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.441399 18486 tablet_bootstrap.cc:492] T 5fa65337d4dc48df91409554d00f5f17 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:38.441401 18526 ts_tablet_manager.cc:1398] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.441860 18526 tablet_bootstrap.cc:492] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:38.442390 18486 tablet_bootstrap.cc:654] T 5fa65337d4dc48df91409554d00f5f17 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.448230 18529 tablet_bootstrap.cc:492] T 5fa65337d4dc48df91409554d00f5f17 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:38.448321 6745 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet 1480785341e049a3b163c3bc97b4736b (table rebalancer_dynamic_table [id=2334ae647b314f1bb0417bec0dc5b86a]). Expected version 1 got 0
I20240503 07:59:38.448747 6741 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 1480785341e049a3b163c3bc97b4736b (table rebalancer_dynamic_table [id=2334ae647b314f1bb0417bec0dc5b86a]). Expected version 1 got 0
I20240503 07:59:38.449563 18488 tablet_bootstrap.cc:492] T 801d28d7a82549b1ad71fbc5e939c261 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:38.449674 18488 ts_tablet_manager.cc:1367] T 801d28d7a82549b1ad71fbc5e939c261 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240503 07:59:38.449741 18529 tablet_bootstrap.cc:654] T 5fa65337d4dc48df91409554d00f5f17 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.450232 18488 raft_consensus.cc:348] T 801d28d7a82549b1ad71fbc5e939c261 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.450409 18488 raft_consensus.cc:374] T 801d28d7a82549b1ad71fbc5e939c261 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.450448 18488 raft_consensus.cc:729] T 801d28d7a82549b1ad71fbc5e939c261 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.450567 18488 consensus_queue.cc:260] T 801d28d7a82549b1ad71fbc5e939c261 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.450661 18532 tablet_bootstrap.cc:492] T 664327fac0a24b95aa459d16efe2ef43 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:38.450874 18488 ts_tablet_manager.cc:1398] T 801d28d7a82549b1ad71fbc5e939c261 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:38.451752 18532 tablet_bootstrap.cc:654] T 664327fac0a24b95aa459d16efe2ef43 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.452253 18526 tablet_bootstrap.cc:654] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.453428 18529 tablet_bootstrap.cc:492] T 5fa65337d4dc48df91409554d00f5f17 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:38.453536 18529 ts_tablet_manager.cc:1367] T 5fa65337d4dc48df91409554d00f5f17 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:38.455433 18529 raft_consensus.cc:348] T 5fa65337d4dc48df91409554d00f5f17 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:38.455744 18529 raft_consensus.cc:374] T 5fa65337d4dc48df91409554d00f5f17 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.456249 18529 raft_consensus.cc:729] T 5fa65337d4dc48df91409554d00f5f17 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.456410 18529 consensus_queue.cc:260] T 5fa65337d4dc48df91409554d00f5f17 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:38.458180 18526 tablet_bootstrap.cc:492] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:38.458359 18526 ts_tablet_manager.cc:1367] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 07:59:38.458971 18526 raft_consensus.cc:348] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:38.459158 18526 raft_consensus.cc:374] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.459231 18526 raft_consensus.cc:729] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.459415 18526 consensus_queue.cc:260] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:38.461238 18526 ts_tablet_manager.cc:1398] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:38.462155 18527 tablet_bootstrap.cc:492] T 664327fac0a24b95aa459d16efe2ef43 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:38.462607 18529 ts_tablet_manager.cc:1398] T 5fa65337d4dc48df91409554d00f5f17 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:38.462924 18529 tablet_bootstrap.cc:492] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:38.463454 18527 ts_tablet_manager.cc:1367] T 664327fac0a24b95aa459d16efe2ef43 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.001s
I20240503 07:59:38.464123 18527 raft_consensus.cc:348] T 664327fac0a24b95aa459d16efe2ef43 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:38.464367 18527 raft_consensus.cc:374] T 664327fac0a24b95aa459d16efe2ef43 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.464473 18527 raft_consensus.cc:729] T 664327fac0a24b95aa459d16efe2ef43 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.464689 18527 consensus_queue.cc:260] T 664327fac0a24b95aa459d16efe2ef43 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:38.466902 18527 ts_tablet_manager.cc:1398] T 664327fac0a24b95aa459d16efe2ef43 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:38.467139 18527 tablet_bootstrap.cc:492] T 801d28d7a82549b1ad71fbc5e939c261 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:38.468518 18529 tablet_bootstrap.cc:654] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.468523 18342 tablet.cc:1722] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.468850 18342 tablet.cc:1620] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:38.469292 18527 tablet_bootstrap.cc:654] T 801d28d7a82549b1ad71fbc5e939c261 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.471053 18527 tablet_bootstrap.cc:492] T 801d28d7a82549b1ad71fbc5e939c261 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:38.471148 18527 ts_tablet_manager.cc:1367] T 801d28d7a82549b1ad71fbc5e939c261 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:38.471622 18527 raft_consensus.cc:348] T 801d28d7a82549b1ad71fbc5e939c261 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.471745 18527 raft_consensus.cc:374] T 801d28d7a82549b1ad71fbc5e939c261 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.471778 18527 raft_consensus.cc:729] T 801d28d7a82549b1ad71fbc5e939c261 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.471918 18532 tablet_bootstrap.cc:492] T 664327fac0a24b95aa459d16efe2ef43 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:38.471915 18527 consensus_queue.cc:260] T 801d28d7a82549b1ad71fbc5e939c261 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.472015 18532 ts_tablet_manager.cc:1367] T 664327fac0a24b95aa459d16efe2ef43 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 07:59:38.472219 18527 ts_tablet_manager.cc:1398] T 801d28d7a82549b1ad71fbc5e939c261 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:38.472537 18532 raft_consensus.cc:348] T 664327fac0a24b95aa459d16efe2ef43 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:38.472677 18532 raft_consensus.cc:374] T 664327fac0a24b95aa459d16efe2ef43 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.472713 18532 raft_consensus.cc:729] T 664327fac0a24b95aa459d16efe2ef43 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.472834 18532 consensus_queue.cc:260] T 664327fac0a24b95aa459d16efe2ef43 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:38.473115 18532 ts_tablet_manager.cc:1398] T 664327fac0a24b95aa459d16efe2ef43 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.473179 6742 catalog_manager.cc:5576] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435) does not have the latest schema for tablet 1480785341e049a3b163c3bc97b4736b (table rebalancer_dynamic_table [id=2334ae647b314f1bb0417bec0dc5b86a]). Expected version 1 got 0
I20240503 07:59:38.473632 18544 tablet.cc:1722] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.474078 18544 tablet.cc:1620] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:38.475092 18529 tablet_bootstrap.cc:492] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:38.475246 18529 ts_tablet_manager.cc:1367] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:38.475778 18529 raft_consensus.cc:348] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:38.475960 18529 raft_consensus.cc:374] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.476035 18529 raft_consensus.cc:729] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.476229 18529 consensus_queue.cc:260] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:38.477777 18529 ts_tablet_manager.cc:1398] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:38.480315 18143 raft_consensus.cc:482] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.480439 18143 raft_consensus.cc:504] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.480841 18143 leader_election.cc:288] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:38.481508 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801d28d7a82549b1ad71fbc5e939c261" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:38.481634 7278 raft_consensus.cc:2454] T 801d28d7a82549b1ad71fbc5e939c261 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:38.481954 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801d28d7a82549b1ad71fbc5e939c261" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:38.482093 9767 raft_consensus.cc:2454] T 801d28d7a82549b1ad71fbc5e939c261 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:38.482419 6956 leader_election.cc:302] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:38.482640 18143 raft_consensus.cc:2781] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.482694 18143 raft_consensus.cc:482] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.482724 18143 raft_consensus.cc:3037] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.483814 18143 raft_consensus.cc:504] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.484089 6742 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet 1480785341e049a3b163c3bc97b4736b (table rebalancer_dynamic_table [id=2334ae647b314f1bb0417bec0dc5b86a]). Expected version 1 got 0
I20240503 07:59:38.484160 18143 leader_election.cc:288] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:38.484884 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801d28d7a82549b1ad71fbc5e939c261" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:38.485008 9767 raft_consensus.cc:3037] T 801d28d7a82549b1ad71fbc5e939c261 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.484982 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801d28d7a82549b1ad71fbc5e939c261" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:38.485111 7282 raft_consensus.cc:3037] T 801d28d7a82549b1ad71fbc5e939c261 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.486060 7282 raft_consensus.cc:2454] T 801d28d7a82549b1ad71fbc5e939c261 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:38.486060 9767 raft_consensus.cc:2454] T 801d28d7a82549b1ad71fbc5e939c261 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:38.486629 6956 leader_election.cc:302] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:38.486888 17517 raft_consensus.cc:2781] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.486966 17517 raft_consensus.cc:686] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:38.487092 17517 consensus_queue.cc:237] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.488617 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=2334ae647b314f1bb0417bec0dc5b86a] alter complete (version 1)
I20240503 07:59:38.489598 6746 catalog_manager.cc:5484] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:38.490886 18546 tablet.cc:1722] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.491325 18546 tablet.cc:1620] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:38.495206 18486 tablet_bootstrap.cc:492] T 5fa65337d4dc48df91409554d00f5f17 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:38.495325 18486 ts_tablet_manager.cc:1367] T 5fa65337d4dc48df91409554d00f5f17 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.002s
I20240503 07:59:38.495822 18486 raft_consensus.cc:348] T 5fa65337d4dc48df91409554d00f5f17 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:38.495931 18486 raft_consensus.cc:374] T 5fa65337d4dc48df91409554d00f5f17 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.495951 18486 raft_consensus.cc:729] T 5fa65337d4dc48df91409554d00f5f17 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.496058 18486 consensus_queue.cc:260] T 5fa65337d4dc48df91409554d00f5f17 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:38.496418 18486 ts_tablet_manager.cc:1398] T 5fa65337d4dc48df91409554d00f5f17 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.500640 9767 consensus_queue.cc:237] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:38.502163 9357 raft_consensus.cc:1261] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:38.502177 6888 raft_consensus.cc:1261] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:38.502588 17401 consensus_queue.cc:1035] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:38.502838 18418 consensus_queue.cc:1035] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:38.505937 17401 raft_consensus.cc:2932] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:38.506435 9357 raft_consensus.cc:2932] T 7009acabeee34602a3ce7462ed5a2d8a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:38.506772 18548 raft_consensus.cc:482] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.506867 18548 raft_consensus.cc:504] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:38.507357 18548 leader_election.cc:288] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:38.508095 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa65337d4dc48df91409554d00f5f17" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:38.508247 7150 raft_consensus.cc:2454] T 5fa65337d4dc48df91409554d00f5f17 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:38.508203 6888 raft_consensus.cc:2932] T 7009acabeee34602a3ce7462ed5a2d8a P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:38.508630 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa65337d4dc48df91409554d00f5f17" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:38.508744 9357 raft_consensus.cc:2454] T 5fa65337d4dc48df91409554d00f5f17 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:38.509665 6956 leader_election.cc:302] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:38.509868 18548 raft_consensus.cc:2781] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.509920 18548 raft_consensus.cc:482] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.509953 18548 raft_consensus.cc:3037] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.510936 6745 catalog_manager.cc:5484] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:38.511097 18548 raft_consensus.cc:504] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:38.511622 18548 leader_election.cc:288] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:38.511829 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa65337d4dc48df91409554d00f5f17" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:38.511971 7150 raft_consensus.cc:3037] T 5fa65337d4dc48df91409554d00f5f17 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.512338 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa65337d4dc48df91409554d00f5f17" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:38.512446 9357 raft_consensus.cc:3037] T 5fa65337d4dc48df91409554d00f5f17 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.513118 7150 raft_consensus.cc:2454] T 5fa65337d4dc48df91409554d00f5f17 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:38.513500 9357 raft_consensus.cc:2454] T 5fa65337d4dc48df91409554d00f5f17 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:38.513922 6954 leader_election.cc:302] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:38.515647 18548 raft_consensus.cc:2781] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.515869 18548 raft_consensus.cc:686] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:38.516078 18548 consensus_queue.cc:237] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:38.516351 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet 7009acabeee34602a3ce7462ed5a2d8a with delete_type TABLET_DATA_TOMBSTONED (TS 159177b3caeb434baa98e04136d050b2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:38.517391 18555 tablet_replica.cc:330] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:38.517680 18555 raft_consensus.cc:2229] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.517840 18555 raft_consensus.cc:2258] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.518389 18555 ts_tablet_manager.cc:1875] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:38.520622 6745 catalog_manager.cc:5484] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:38.521382 18555 ts_tablet_manager.cc:1888] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:38.521497 18555 log.cc:1196] T 7009acabeee34602a3ce7462ed5a2d8a P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/7009acabeee34602a3ce7462ed5a2d8a
I20240503 07:59:38.522097 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 7009acabeee34602a3ce7462ed5a2d8a (table rebalancer_extra_table_24 [id=7055be84cc0f4e368d98a9e128f121af]) successfully deleted
I20240503 07:59:38.597972 6742 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:38.609104 18342 tablet.cc:1722] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.609509 18342 tablet.cc:1620] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: MemRowSet was empty: no flush needed.
I20240503 07:59:38.613597 18546 tablet.cc:1722] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.613976 18546 tablet.cc:1620] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:38.624778 6745 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=2334ae647b314f1bb0417bec0dc5b86a] alter complete (version 2)
I20240503 07:59:38.629133 18544 tablet.cc:1722] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.629492 18544 tablet.cc:1620] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:38.729547 17804 raft_consensus.cc:482] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.729687 17804 raft_consensus.cc:504] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:38.730639 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "664327fac0a24b95aa459d16efe2ef43" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:38.730792 9766 raft_consensus.cc:2454] T 664327fac0a24b95aa459d16efe2ef43 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:38.730851 17804 leader_election.cc:288] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:38.731230 7087 leader_election.cc:302] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:38.731248 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:38.731459 17804 raft_consensus.cc:2781] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.731557 17804 raft_consensus.cc:482] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.731666 17804 raft_consensus.cc:3037] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.731916 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "664327fac0a24b95aa459d16efe2ef43" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:38.732077 9636 raft_consensus.cc:2454] T 664327fac0a24b95aa459d16efe2ef43 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:38.732918 17804 raft_consensus.cc:504] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:38.733908 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "664327fac0a24b95aa459d16efe2ef43" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:38.733989 17804 leader_election.cc:288] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:38.734068 9766 raft_consensus.cc:3037] T 664327fac0a24b95aa459d16efe2ef43 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.735962 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "664327fac0a24b95aa459d16efe2ef43" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:38.736109 9636 raft_consensus.cc:3037] T 664327fac0a24b95aa459d16efe2ef43 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.737241 9636 raft_consensus.cc:2454] T 664327fac0a24b95aa459d16efe2ef43 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:38.737757 18342 tablet.cc:1722] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.738006 18544 tablet.cc:1722] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.738404 18546 tablet.cc:1722] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.740410 9766 raft_consensus.cc:2454] T 664327fac0a24b95aa459d16efe2ef43 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:38.740803 7087 leader_election.cc:302] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:38.741076 17804 raft_consensus.cc:2781] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.741444 17804 raft_consensus.cc:686] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:38.741650 17804 consensus_queue.cc:237] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:38.744612 6738 catalog_manager.cc:5484] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:38.746564 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=2334ae647b314f1bb0417bec0dc5b86a] alter complete (version 3)
I20240503 07:59:38.764142 9615 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:38.774057 6998 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:38.777175 9350 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:38.779882 7129 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:38.783366 9747 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:38.792614 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:38.813999 9483 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:38.835474 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:38.849169 6741 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:38.849340 6741 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:38.851562 6741 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 1480785341e049a3b163c3bc97b4736b
I20240503 07:59:38.852406 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet 1480785341e049a3b163c3bc97b4736b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:38 UTC) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:38.852473 9483 tablet_service.cc:1514] Processing DeleteTablet for tablet 1480785341e049a3b163c3bc97b4736b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:38 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:38.852607 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet 1480785341e049a3b163c3bc97b4736b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:38 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:38.853096 18610 tablet_replica.cc:330] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:38.853322 18610 raft_consensus.cc:2229] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:38.853577 18610 raft_consensus.cc:2258] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.854255 18610 ts_tablet_manager.cc:1875] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:38.854396 18612 tablet_replica.cc:330] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:38.854605 18612 raft_consensus.cc:2229] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.854827 18612 raft_consensus.cc:2258] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.855594 18612 ts_tablet_manager.cc:1875] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:38.856976 18610 ts_tablet_manager.cc:1888] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:38.857065 18610 log.cc:1196] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/1480785341e049a3b163c3bc97b4736b
I20240503 07:59:38.857319 18610 ts_tablet_manager.cc:1909] T 1480785341e049a3b163c3bc97b4736b P 3befbb2d206f4be89d3e07f38837374b: Deleting consensus metadata
I20240503 07:59:38.857970 18612 ts_tablet_manager.cc:1888] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:38.858044 18611 tablet_replica.cc:330] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:38.858080 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 1480785341e049a3b163c3bc97b4736b (table rebalancer_dynamic_table_new_name [id=2334ae647b314f1bb0417bec0dc5b86a]) successfully deleted
I20240503 07:59:38.858202 18611 raft_consensus.cc:2229] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.858330 18611 raft_consensus.cc:2258] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.858888 18611 ts_tablet_manager.cc:1875] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:38.860522 18612 log.cc:1196] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/1480785341e049a3b163c3bc97b4736b
I20240503 07:59:38.860908 18612 ts_tablet_manager.cc:1909] T 1480785341e049a3b163c3bc97b4736b P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:38.861742 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 1480785341e049a3b163c3bc97b4736b (table rebalancer_dynamic_table_new_name [id=2334ae647b314f1bb0417bec0dc5b86a]) successfully deleted
I20240503 07:59:38.862535 18611 ts_tablet_manager.cc:1888] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:38.862608 18611 log.cc:1196] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/1480785341e049a3b163c3bc97b4736b
I20240503 07:59:38.862844 18611 ts_tablet_manager.cc:1909] T 1480785341e049a3b163c3bc97b4736b P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:38.863551 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 1480785341e049a3b163c3bc97b4736b (table rebalancer_dynamic_table_new_name [id=2334ae647b314f1bb0417bec0dc5b86a]) successfully deleted
I20240503 07:59:38.886852 9636 consensus_queue.cc:237] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:38.888177 9766 raft_consensus.cc:1261] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:38.888538 7020 raft_consensus.cc:1261] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:38.889042 17297 consensus_queue.cc:1035] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:38.889276 18513 consensus_queue.cc:1035] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:38.891621 17297 raft_consensus.cc:2932] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:38.892748 9766 raft_consensus.cc:2932] T d05ebb7c3aa24804a7aa9a7e1a8e699c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:38.893056 7020 raft_consensus.cc:2932] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:38.896551 6741 catalog_manager.cc:5484] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 2 to 3, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:38.900434 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet d05ebb7c3aa24804a7aa9a7e1a8e699c with delete_type TABLET_DATA_TOMBSTONED (TS 00a3f8e159c64327832d79fb32cdcca8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:38.900796 18636 tablet_replica.cc:330] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:38.900998 18636 raft_consensus.cc:2229] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.901178 18636 raft_consensus.cc:2258] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.901827 18636 ts_tablet_manager.cc:1875] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:38.904409 18636 ts_tablet_manager.cc:1888] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:38.904489 18636 log.cc:1196] T d05ebb7c3aa24804a7aa9a7e1a8e699c P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d05ebb7c3aa24804a7aa9a7e1a8e699c
I20240503 07:59:38.905349 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet d05ebb7c3aa24804a7aa9a7e1a8e699c (table rebalancer_extra_table_32 [id=62360cc567504bf19c32fc20e93a496f]) successfully deleted
I20240503 07:59:38.942131 18143 consensus_queue.cc:1035] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.947860 18548 consensus_queue.cc:1035] T 801d28d7a82549b1ad71fbc5e939c261 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.953382 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:38.963279 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 032346eec4ad44038d4f496b6f4f98d9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f400170777c14a0183f58b20055489d7]), partition=
I20240503 07:59:38.963626 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032346eec4ad44038d4f496b6f4f98d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.965364 18143 consensus_queue.cc:1035] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.966542 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 032346eec4ad44038d4f496b6f4f98d9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f400170777c14a0183f58b20055489d7]), partition=
I20240503 07:59:38.966827 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032346eec4ad44038d4f496b6f4f98d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.969146 18527 tablet_bootstrap.cc:492] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:38.969342 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 032346eec4ad44038d4f496b6f4f98d9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f400170777c14a0183f58b20055489d7]), partition=
I20240503 07:59:38.969568 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032346eec4ad44038d4f496b6f4f98d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.970126 18527 tablet_bootstrap.cc:654] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.971670 18527 tablet_bootstrap.cc:492] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:38.971789 18527 ts_tablet_manager.cc:1367] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:38.972307 18527 raft_consensus.cc:348] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.972429 18527 raft_consensus.cc:374] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.972445 18532 tablet_bootstrap.cc:492] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:38.972462 18527 raft_consensus.cc:729] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.972612 18527 consensus_queue.cc:260] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.973453 18532 tablet_bootstrap.cc:654] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.975100 18527 ts_tablet_manager.cc:1398] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:38.980933 18143 consensus_queue.cc:1035] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.982494 18532 tablet_bootstrap.cc:492] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:38.982637 18532 ts_tablet_manager.cc:1367] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:38.983180 18532 raft_consensus.cc:348] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.983307 18532 raft_consensus.cc:374] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.983340 18532 raft_consensus.cc:729] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.983460 18532 consensus_queue.cc:260] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:38.983755 18532 ts_tablet_manager.cc:1398] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.997316 18677 tablet_bootstrap.cc:492] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:38.999266 18677 tablet_bootstrap.cc:654] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.006639 18677 tablet_bootstrap.cc:492] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:39.006879 18677 ts_tablet_manager.cc:1367] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240503 07:59:39.007479 18677 raft_consensus.cc:348] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:39.007680 18677 raft_consensus.cc:374] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.007747 18677 raft_consensus.cc:729] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.007973 18677 consensus_queue.cc:260] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:39.008349 18677 ts_tablet_manager.cc:1398] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:39.032289 18418 raft_consensus.cc:482] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.032439 18418 raft_consensus.cc:504] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:39.032938 18418 leader_election.cc:288] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:39.034406 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032346eec4ad44038d4f496b6f4f98d9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:39.034443 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032346eec4ad44038d4f496b6f4f98d9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:39.034577 6888 raft_consensus.cc:2454] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:39.034626 9636 raft_consensus.cc:2454] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:39.035003 9700 leader_election.cc:302] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:39.035208 18418 raft_consensus.cc:2781] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.035264 18418 raft_consensus.cc:482] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.035300 18418 raft_consensus.cc:3037] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.036649 18418 raft_consensus.cc:504] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:39.037053 18418 leader_election.cc:288] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:39.037492 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032346eec4ad44038d4f496b6f4f98d9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:39.037628 6888 raft_consensus.cc:3037] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.038064 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032346eec4ad44038d4f496b6f4f98d9" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:39.038177 9636 raft_consensus.cc:3037] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.038931 6888 raft_consensus.cc:2454] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:39.039292 9636 raft_consensus.cc:2454] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:39.041743 9700 leader_election.cc:302] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:39.041949 18418 raft_consensus.cc:2781] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.042053 18418 raft_consensus.cc:686] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:39.042191 18418 consensus_queue.cc:237] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:39.044993 6738 catalog_manager.cc:5484] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:39.097599 9350 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:39.098693 7129 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:39.100716 6998 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:39.105358 9615 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:39.106545 9483 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:39.115242 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:39.123373 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:39.135032 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:39.146659 17804 consensus_queue.cc:1035] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.151029 17340 consensus_queue.cc:1035] T 664327fac0a24b95aa459d16efe2ef43 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.152940 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:39.159358 9636 raft_consensus.cc:1261] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.159891 6886 raft_consensus.cc:1261] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.159955 17401 consensus_queue.cc:1035] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.160487 17401 consensus_queue.cc:1035] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.164702 6738 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 032346eec4ad44038d4f496b6f4f98d9 (table rebalancer_dynamic_table [id=f400170777c14a0183f58b20055489d7]). Expected version 1 got 0
I20240503 07:59:39.166000 6745 catalog_manager.cc:5576] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655) does not have the latest schema for tablet 032346eec4ad44038d4f496b6f4f98d9 (table rebalancer_dynamic_table [id=f400170777c14a0183f58b20055489d7]). Expected version 1 got 0
I20240503 07:59:39.167168 6738 catalog_manager.cc:5576] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369) does not have the latest schema for tablet 032346eec4ad44038d4f496b6f4f98d9 (table rebalancer_dynamic_table [id=f400170777c14a0183f58b20055489d7]). Expected version 1 got 0
I20240503 07:59:39.178750 18705 tablet.cc:1722] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:39.179165 18705 tablet.cc:1620] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:39.183964 18704 tablet.cc:1722] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:39.184911 18704 tablet.cc:1620] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:39.185990 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=f400170777c14a0183f58b20055489d7] alter complete (version 1)
I20240503 07:59:39.193069 18706 tablet.cc:1722] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:39.193557 18706 tablet.cc:1620] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:39.194638 6738 catalog_manager.cc:5576] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655) does not have the latest schema for tablet 032346eec4ad44038d4f496b6f4f98d9 (table rebalancer_dynamic_table [id=f400170777c14a0183f58b20055489d7]). Expected version 1 got 0
I20240503 07:59:39.229221 7150 consensus_queue.cc:237] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.230603 9357 raft_consensus.cc:1261] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:39.230603 9504 raft_consensus.cc:1261] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:39.231154 18697 consensus_queue.cc:1035] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:39.232462 18697 consensus_queue.cc:1035] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:39.233589 18697 raft_consensus.cc:2932] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:39.235020 9504 raft_consensus.cc:2932] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:39.235661 9357 raft_consensus.cc:2932] T 5390e70b0449449aa3af05112bd29797 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:39.239259 6738 catalog_manager.cc:5484] T 5390e70b0449449aa3af05112bd29797 P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 6 to 7, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:39.242622 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet 5390e70b0449449aa3af05112bd29797 with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:39.242884 18612 tablet_replica.cc:330] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:39.243038 18612 raft_consensus.cc:2229] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.243217 18612 raft_consensus.cc:2258] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.243818 18612 ts_tablet_manager.cc:1875] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:39.245874 18612 ts_tablet_manager.cc:1888] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 07:59:39.245958 18612 log.cc:1196] T 5390e70b0449449aa3af05112bd29797 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5390e70b0449449aa3af05112bd29797
I20240503 07:59:39.246467 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 5390e70b0449449aa3af05112bd29797 (table rebalance_test_table_7 [id=354bc6a2b3d54351a5a87e662220aaca]) successfully deleted
I20240503 07:59:39.263988 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:39.272389 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 6bbf3e180fcf4d7ab56f7b7e967ab7bc (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=46b1872433a04bd6bb9e859e3e020f02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.272784 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bbf3e180fcf4d7ab56f7b7e967ab7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.273221 9483 tablet_service.cc:1467] Processing CreateTablet for tablet d66715134ff740129267a5300af8cb19 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=46b1872433a04bd6bb9e859e3e020f02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.273491 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66715134ff740129267a5300af8cb19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.273509 7260 tablet_service.cc:1467] Processing CreateTablet for tablet e33d88b13ce04b18876d4ce66b435d34 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=46b1872433a04bd6bb9e859e3e020f02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.273775 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33d88b13ce04b18876d4ce66b435d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.275980 7261 tablet_service.cc:1467] Processing CreateTablet for tablet d66715134ff740129267a5300af8cb19 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=46b1872433a04bd6bb9e859e3e020f02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.276073 9484 tablet_service.cc:1467] Processing CreateTablet for tablet e33d88b13ce04b18876d4ce66b435d34 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=46b1872433a04bd6bb9e859e3e020f02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.276235 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66715134ff740129267a5300af8cb19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.276330 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33d88b13ce04b18876d4ce66b435d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.278520 9350 tablet_service.cc:1467] Processing CreateTablet for tablet d66715134ff740129267a5300af8cb19 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=46b1872433a04bd6bb9e859e3e020f02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.278843 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66715134ff740129267a5300af8cb19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.280505 18737 tablet_bootstrap.cc:492] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:39.281172 18739 tablet_bootstrap.cc:492] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:39.281903 9747 tablet_service.cc:1467] Processing CreateTablet for tablet e33d88b13ce04b18876d4ce66b435d34 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=46b1872433a04bd6bb9e859e3e020f02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.282182 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33d88b13ce04b18876d4ce66b435d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.282384 18739 tablet_bootstrap.cc:654] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.283375 18737 tablet_bootstrap.cc:654] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.283833 18739 tablet_bootstrap.cc:492] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:39.283918 18739 ts_tablet_manager.cc:1367] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:39.284091 18738 tablet_bootstrap.cc:492] T e33d88b13ce04b18876d4ce66b435d34 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:39.284405 18739 raft_consensus.cc:348] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.284523 18739 raft_consensus.cc:374] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.284555 18739 raft_consensus.cc:729] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.284564 18527 tablet_bootstrap.cc:492] T e33d88b13ce04b18876d4ce66b435d34 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:39.281177 9351 tablet_service.cc:1467] Processing CreateTablet for tablet 6bbf3e180fcf4d7ab56f7b7e967ab7bc (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=46b1872433a04bd6bb9e859e3e020f02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.284674 18739 consensus_queue.cc:260] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.284883 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bbf3e180fcf4d7ab56f7b7e967ab7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.284962 18739 ts_tablet_manager.cc:1398] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.285027 18738 tablet_bootstrap.cc:654] T e33d88b13ce04b18876d4ce66b435d34 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.285539 18527 tablet_bootstrap.cc:654] T e33d88b13ce04b18876d4ce66b435d34 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.286024 18737 tablet_bootstrap.cc:492] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:39.286127 18737 ts_tablet_manager.cc:1367] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:39.286881 18737 raft_consensus.cc:348] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.287026 18737 raft_consensus.cc:374] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.287065 18737 raft_consensus.cc:729] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.287235 18737 consensus_queue.cc:260] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.287552 18737 ts_tablet_manager.cc:1398] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.288233 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 6bbf3e180fcf4d7ab56f7b7e967ab7bc (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=46b1872433a04bd6bb9e859e3e020f02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.288533 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bbf3e180fcf4d7ab56f7b7e967ab7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.289062 6745 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:39.291838 18739 tablet_bootstrap.cc:492] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:39.292948 18737 tablet_bootstrap.cc:492] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:39.293115 18739 tablet_bootstrap.cc:654] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 3befbb2d206f4be89d3e07f38837374b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.294121 18527 tablet_bootstrap.cc:492] T e33d88b13ce04b18876d4ce66b435d34 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:39.294181 18737 tablet_bootstrap.cc:654] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.294250 18527 ts_tablet_manager.cc:1367] T e33d88b13ce04b18876d4ce66b435d34 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240503 07:59:39.294824 18527 raft_consensus.cc:348] T e33d88b13ce04b18876d4ce66b435d34 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.294957 18527 raft_consensus.cc:374] T e33d88b13ce04b18876d4ce66b435d34 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.294991 18527 raft_consensus.cc:729] T e33d88b13ce04b18876d4ce66b435d34 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.295125 18527 consensus_queue.cc:260] T e33d88b13ce04b18876d4ce66b435d34 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.295424 18527 ts_tablet_manager.cc:1398] T e33d88b13ce04b18876d4ce66b435d34 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.296321 18738 tablet_bootstrap.cc:492] T e33d88b13ce04b18876d4ce66b435d34 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:39.296475 18738 ts_tablet_manager.cc:1367] T e33d88b13ce04b18876d4ce66b435d34 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:39.297011 18739 tablet_bootstrap.cc:492] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 3befbb2d206f4be89d3e07f38837374b: No bootstrap required, opened a new log
I20240503 07:59:39.297106 18739 ts_tablet_manager.cc:1367] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:39.297580 18739 raft_consensus.cc:348] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:39.297713 18739 raft_consensus.cc:374] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.297745 18739 raft_consensus.cc:729] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.297871 18739 consensus_queue.cc:260] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:39.298141 18739 ts_tablet_manager.cc:1398] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.298509 18738 raft_consensus.cc:348] T e33d88b13ce04b18876d4ce66b435d34 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.298650 18738 raft_consensus.cc:374] T e33d88b13ce04b18876d4ce66b435d34 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.298682 18738 raft_consensus.cc:729] T e33d88b13ce04b18876d4ce66b435d34 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.298807 18738 consensus_queue.cc:260] T e33d88b13ce04b18876d4ce66b435d34 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.299582 18738 ts_tablet_manager.cc:1398] T e33d88b13ce04b18876d4ce66b435d34 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:39.299924 18738 tablet_bootstrap.cc:492] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:39.301278 18738 tablet_bootstrap.cc:654] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.303265 18737 tablet_bootstrap.cc:492] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:39.303364 18737 ts_tablet_manager.cc:1367] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:39.303675 18704 tablet.cc:1722] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:39.303998 18704 tablet.cc:1620] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: MemRowSet was empty: no flush needed.
I20240503 07:59:39.304235 18737 raft_consensus.cc:348] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.304359 18737 raft_consensus.cc:374] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.304391 18737 raft_consensus.cc:729] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.304515 18737 consensus_queue.cc:260] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.305707 18706 tablet.cc:1722] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:39.306056 18706 tablet.cc:1620] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: MemRowSet was empty: no flush needed.
I20240503 07:59:39.307907 18737 ts_tablet_manager.cc:1398] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:39.308883 18705 tablet.cc:1722] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:39.309192 18705 tablet.cc:1620] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: MemRowSet was empty: no flush needed.
I20240503 07:59:39.310153 18736 tablet_bootstrap.cc:492] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:39.310300 18751 tablet_bootstrap.cc:492] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:39.314134 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=f400170777c14a0183f58b20055489d7] alter complete (version 2)
I20240503 07:59:39.316217 18738 tablet_bootstrap.cc:492] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:39.317122 18738 ts_tablet_manager.cc:1367] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 07:59:39.317678 18738 raft_consensus.cc:348] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.318219 18738 raft_consensus.cc:374] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.318325 18738 raft_consensus.cc:729] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.318384 18751 tablet_bootstrap.cc:654] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.318557 18738 consensus_queue.cc:260] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.318907 18738 ts_tablet_manager.cc:1398] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:39.319180 18736 tablet_bootstrap.cc:654] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.322471 18736 tablet_bootstrap.cc:492] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:39.322620 18751 tablet_bootstrap.cc:492] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:39.322732 18736 ts_tablet_manager.cc:1367] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:39.322782 18751 ts_tablet_manager.cc:1367] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:39.323263 18751 raft_consensus.cc:348] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:39.323269 18736 raft_consensus.cc:348] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:39.323372 18751 raft_consensus.cc:374] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.323379 18736 raft_consensus.cc:374] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.323402 18751 raft_consensus.cc:729] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.323407 18736 raft_consensus.cc:729] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.323541 18736 consensus_queue.cc:260] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:39.323572 18751 consensus_queue.cc:260] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:39.323835 18736 ts_tablet_manager.cc:1398] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:39.325016 18751 ts_tablet_manager.cc:1398] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:39.350814 18264 raft_consensus.cc:482] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.350960 18264 raft_consensus.cc:504] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.351584 18264 leader_election.cc:288] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:39.352476 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66715134ff740129267a5300af8cb19" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:39.352685 7278 raft_consensus.cc:2454] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:39.353619 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66715134ff740129267a5300af8cb19" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:39.353797 9504 raft_consensus.cc:2454] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 0.
I20240503 07:59:39.355916 9310 leader_election.cc:302] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:39.356369 18524 raft_consensus.cc:2781] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.356438 18524 raft_consensus.cc:482] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.356477 18524 raft_consensus.cc:3037] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.358740 18524 raft_consensus.cc:504] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.359208 18524 leader_election.cc:288] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Requested vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:39.359716 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66715134ff740129267a5300af8cb19" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:39.359853 7278 raft_consensus.cc:3037] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.359846 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66715134ff740129267a5300af8cb19" candidate_uuid: "3befbb2d206f4be89d3e07f38837374b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:39.359969 9504 raft_consensus.cc:3037] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.361032 7278 raft_consensus.cc:2454] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:39.361032 9504 raft_consensus.cc:2454] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3befbb2d206f4be89d3e07f38837374b in term 1.
I20240503 07:59:39.361397 9310 leader_election.cc:302] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3befbb2d206f4be89d3e07f38837374b, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:39.361608 18524 raft_consensus.cc:2781] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.361701 18524 raft_consensus.cc:686] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Leader. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:39.361832 18524 consensus_queue.cc:237] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.366587 6742 catalog_manager.cc:5484] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3befbb2d206f4be89d3e07f38837374b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:39.415732 6742 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:39.423007 18704 tablet.cc:1722] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:39.425207 18705 tablet.cc:1722] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:39.433684 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=f400170777c14a0183f58b20055489d7] alter complete (version 3)
I20240503 07:59:39.434841 18706 tablet.cc:1722] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:39.444968 17340 raft_consensus.cc:482] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.445111 17340 raft_consensus.cc:504] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:39.445621 17340 leader_election.cc:288] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:39.445940 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bbf3e180fcf4d7ab56f7b7e967ab7bc" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:39.446084 7019 raft_consensus.cc:2454] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:39.446064 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bbf3e180fcf4d7ab56f7b7e967ab7bc" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b" is_pre_election: true
I20240503 07:59:39.446229 9357 raft_consensus.cc:2454] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:39.446444 7086 leader_election.cc:302] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:39.446604 17340 raft_consensus.cc:2781] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.446663 17340 raft_consensus.cc:482] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.446694 17340 raft_consensus.cc:3037] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.447873 17340 raft_consensus.cc:504] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:39.448302 17340 leader_election.cc:288] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:39.448596 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bbf3e180fcf4d7ab56f7b7e967ab7bc" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:39.448726 7019 raft_consensus.cc:3037] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.449882 7019 raft_consensus.cc:2454] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:39.449875 9373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bbf3e180fcf4d7ab56f7b7e967ab7bc" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:39.450023 9373 raft_consensus.cc:3037] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 3befbb2d206f4be89d3e07f38837374b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.450960 7086 leader_election.cc:302] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:39.451193 9373 raft_consensus.cc:2454] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:39.451279 18697 raft_consensus.cc:2781] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.451372 18697 raft_consensus.cc:686] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:39.451516 18697 consensus_queue.cc:237] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:39.451799 18139 raft_consensus.cc:482] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.451905 18139 raft_consensus.cc:504] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.452327 18139 leader_election.cc:288] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:39.452895 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33d88b13ce04b18876d4ce66b435d34" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:39.453042 9767 raft_consensus.cc:2454] T e33d88b13ce04b18876d4ce66b435d34 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:39.453358 9441 leader_election.cc:302] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:39.453444 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33d88b13ce04b18876d4ce66b435d34" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:39.453591 7278 raft_consensus.cc:2454] T e33d88b13ce04b18876d4ce66b435d34 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:39.453778 18139 raft_consensus.cc:2781] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.453836 18139 raft_consensus.cc:482] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.453866 18139 raft_consensus.cc:3037] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.454452 6742 catalog_manager.cc:5484] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:39.455011 18139 raft_consensus.cc:504] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.455389 18139 leader_election.cc:288] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:39.456163 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33d88b13ce04b18876d4ce66b435d34" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:39.456307 7278 raft_consensus.cc:3037] T e33d88b13ce04b18876d4ce66b435d34 P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.457429 7278 raft_consensus.cc:2454] T e33d88b13ce04b18876d4ce66b435d34 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:39.458535 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33d88b13ce04b18876d4ce66b435d34" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:39.458536 9441 leader_election.cc:302] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:39.458678 9767 raft_consensus.cc:3037] T e33d88b13ce04b18876d4ce66b435d34 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.458969 17600 raft_consensus.cc:2781] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.459046 17600 raft_consensus.cc:686] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:39.459184 17600 consensus_queue.cc:237] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:39.459719 9767 raft_consensus.cc:2454] T e33d88b13ce04b18876d4ce66b435d34 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:39.462698 6742 catalog_manager.cc:5484] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:39.484300 7129 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:39.491608 6998 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:39.493371 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:39.500675 9615 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:39.503137 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:39.507607 9351 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:39.530544 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:39.532195 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:39.537778 6742 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:39.537950 6742 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:39.540745 6742 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet 032346eec4ad44038d4f496b6f4f98d9
I20240503 07:59:39.541666 9747 tablet_service.cc:1514] Processing DeleteTablet for tablet 032346eec4ad44038d4f496b6f4f98d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:39 UTC) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:39.542243 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet 032346eec4ad44038d4f496b6f4f98d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:39 UTC) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:39.542546 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet 032346eec4ad44038d4f496b6f4f98d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:39 UTC) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:39.548439 18796 tablet_replica.cc:330] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:39.548707 18796 raft_consensus.cc:2229] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.548916 18796 raft_consensus.cc:2258] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.549845 18796 ts_tablet_manager.cc:1875] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:39.557399 18796 ts_tablet_manager.cc:1888] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:39.557659 18796 log.cc:1196] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/032346eec4ad44038d4f496b6f4f98d9
I20240503 07:59:39.558248 18796 ts_tablet_manager.cc:1909] T 032346eec4ad44038d4f496b6f4f98d9 P 159177b3caeb434baa98e04136d050b2: Deleting consensus metadata
I20240503 07:59:39.559441 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 032346eec4ad44038d4f496b6f4f98d9 (table rebalancer_dynamic_table_new_name [id=f400170777c14a0183f58b20055489d7]) successfully deleted
I20240503 07:59:39.569938 18797 tablet_replica.cc:330] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:39.570135 18797 raft_consensus.cc:2229] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.570278 18797 raft_consensus.cc:2258] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.570557 18795 tablet_replica.cc:330] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:39.570763 18795 raft_consensus.cc:2229] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:39.571022 18795 raft_consensus.cc:2258] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.571137 18797 ts_tablet_manager.cc:1875] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:39.571683 18795 ts_tablet_manager.cc:1875] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:39.573917 18795 ts_tablet_manager.cc:1888] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:39.573916 18797 ts_tablet_manager.cc:1888] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:39.574000 18797 log.cc:1196] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/032346eec4ad44038d4f496b6f4f98d9
I20240503 07:59:39.574065 18795 log.cc:1196] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/032346eec4ad44038d4f496b6f4f98d9
I20240503 07:59:39.574373 18795 ts_tablet_manager.cc:1909] T 032346eec4ad44038d4f496b6f4f98d9 P a372e98a670f49e9adca46ab5c836c4a: Deleting consensus metadata
I20240503 07:59:39.574394 18797 ts_tablet_manager.cc:1909] T 032346eec4ad44038d4f496b6f4f98d9 P c4218904dd7e42d2be95b901a6e5db92: Deleting consensus metadata
I20240503 07:59:39.575299 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 032346eec4ad44038d4f496b6f4f98d9 (table rebalancer_dynamic_table_new_name [id=f400170777c14a0183f58b20055489d7]) successfully deleted
I20240503 07:59:39.575501 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 032346eec4ad44038d4f496b6f4f98d9 (table rebalancer_dynamic_table_new_name [id=f400170777c14a0183f58b20055489d7]) successfully deleted
I20240503 07:59:39.603577 9373 consensus_queue.cc:237] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:39.604892 7150 raft_consensus.cc:1261] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:39.604897 7019 raft_consensus.cc:1261] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:39.605386 18264 consensus_queue.cc:1035] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:39.605576 18794 consensus_queue.cc:1035] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:39.607507 18524 raft_consensus.cc:2932] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:39.607985 7019 raft_consensus.cc:2932] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:39.608187 7150 raft_consensus.cc:2932] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:39.612058 6739 catalog_manager.cc:5484] T 4ef6defd02954e009565bae98f5e8418 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:39.615981 9747 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ef6defd02954e009565bae98f5e8418 with delete_type TABLET_DATA_TOMBSTONED (TS a372e98a670f49e9adca46ab5c836c4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:39.616214 18795 tablet_replica.cc:330] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:39.616358 18795 raft_consensus.cc:2229] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.616477 18795 raft_consensus.cc:2258] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.616950 18795 ts_tablet_manager.cc:1875] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:39.619251 18795 ts_tablet_manager.cc:1888] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:39.619334 18795 log.cc:1196] T 4ef6defd02954e009565bae98f5e8418 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/4ef6defd02954e009565bae98f5e8418
I20240503 07:59:39.620095 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 4ef6defd02954e009565bae98f5e8418 (table rebalancer_extra_table_30 [id=ae9c8bb841ff4e139b8fc59887fda0e9]) successfully deleted
I20240503 07:59:39.642690 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:39.649480 9484 tablet_service.cc:1467] Processing CreateTablet for tablet c9eef3b835ac41dfa3a2054128d7a14f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=307cb381333441b0978242590eb0564b]), partition=
I20240503 07:59:39.649502 6998 tablet_service.cc:1467] Processing CreateTablet for tablet c9eef3b835ac41dfa3a2054128d7a14f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=307cb381333441b0978242590eb0564b]), partition=
I20240503 07:59:39.649789 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9eef3b835ac41dfa3a2054128d7a14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.649799 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9eef3b835ac41dfa3a2054128d7a14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.652217 18737 tablet_bootstrap.cc:492] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:39.652299 18736 tablet_bootstrap.cc:492] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:39.653210 18737 tablet_bootstrap.cc:654] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.653228 18736 tablet_bootstrap.cc:654] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.655687 7261 tablet_service.cc:1467] Processing CreateTablet for tablet c9eef3b835ac41dfa3a2054128d7a14f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=307cb381333441b0978242590eb0564b]), partition=
I20240503 07:59:39.655983 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9eef3b835ac41dfa3a2054128d7a14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.656606 18737 tablet_bootstrap.cc:492] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:39.656703 18737 ts_tablet_manager.cc:1367] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:39.657271 18737 raft_consensus.cc:348] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:39.657394 18737 raft_consensus.cc:374] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.657424 18737 raft_consensus.cc:729] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.657552 18737 consensus_queue.cc:260] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:39.657855 18737 ts_tablet_manager.cc:1398] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.658804 18738 tablet_bootstrap.cc:492] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:39.659852 18738 tablet_bootstrap.cc:654] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.665066 18736 tablet_bootstrap.cc:492] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:39.665117 18738 tablet_bootstrap.cc:492] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:39.665199 18736 ts_tablet_manager.cc:1367] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:39.665212 18738 ts_tablet_manager.cc:1367] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:39.665769 18736 raft_consensus.cc:348] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:39.665892 18736 raft_consensus.cc:374] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.665802 18738 raft_consensus.cc:348] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:39.665912 18736 raft_consensus.cc:729] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.665931 18738 raft_consensus.cc:374] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.665961 18738 raft_consensus.cc:729] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.666033 18736 consensus_queue.cc:260] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:39.666132 18738 consensus_queue.cc:260] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:39.666384 18736 ts_tablet_manager.cc:1398] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.666868 18738 ts_tablet_manager.cc:1398] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:39.725697 17600 raft_consensus.cc:482] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.725870 17600 raft_consensus.cc:504] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:39.726269 17600 leader_election.cc:288] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:39.726670 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9eef3b835ac41dfa3a2054128d7a14f" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:39.726711 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9eef3b835ac41dfa3a2054128d7a14f" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:39.726825 7019 raft_consensus.cc:2454] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:39.726823 7278 raft_consensus.cc:2454] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 0.
I20240503 07:59:39.727214 9441 leader_election.cc:302] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:39.728502 17600 raft_consensus.cc:2781] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.728572 17600 raft_consensus.cc:482] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.728608 17600 raft_consensus.cc:3037] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.729830 17600 raft_consensus.cc:504] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:39.730213 17600 leader_election.cc:288] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Requested vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:39.730676 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9eef3b835ac41dfa3a2054128d7a14f" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:39.730821 7278 raft_consensus.cc:3037] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.730918 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9eef3b835ac41dfa3a2054128d7a14f" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:39.731041 7019 raft_consensus.cc:3037] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.731866 7278 raft_consensus.cc:2454] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:39.732138 7019 raft_consensus.cc:2454] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:39.732616 9440 leader_election.cc:302] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:39.732910 18139 raft_consensus.cc:2781] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.732976 18139 raft_consensus.cc:686] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Becoming Leader. State: Replica: cb73994515af4880b05383e78368bb4a, State: Running, Role: LEADER
I20240503 07:59:39.733094 18139 consensus_queue.cc:237] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:39.736584 6742 catalog_manager.cc:5484] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb73994515af4880b05383e78368bb4a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:39.744505 18524 consensus_queue.cc:1035] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.749135 18264 consensus_queue.cc:1035] T d66715134ff740129267a5300af8cb19 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.790532 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:39.791076 6998 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:39.793174 9615 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:39.797298 9351 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:39.802320 7129 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:39.821795 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:39.832609 17340 consensus_queue.cc:1035] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.843326 9747 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:39.843895 9484 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:39.850903 18697 consensus_queue.cc:1035] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.852820 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:39.861093 7019 raft_consensus.cc:1261] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.862112 7278 raft_consensus.cc:1261] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.863536 18865 consensus_queue.cc:1035] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.864113 17600 consensus_queue.cc:1035] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.867914 6738 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet c9eef3b835ac41dfa3a2054128d7a14f (table rebalancer_dynamic_table [id=307cb381333441b0978242590eb0564b]). Expected version 1 got 0
I20240503 07:59:39.869099 6745 catalog_manager.cc:5576] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383) does not have the latest schema for tablet c9eef3b835ac41dfa3a2054128d7a14f (table rebalancer_dynamic_table [id=307cb381333441b0978242590eb0564b]). Expected version 1 got 0
I20240503 07:59:39.877130 18891 tablet.cc:1722] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:39.877681 18891 tablet.cc:1620] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:39.878141 6745 catalog_manager.cc:5576] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383) does not have the latest schema for tablet c9eef3b835ac41dfa3a2054128d7a14f (table rebalancer_dynamic_table [id=307cb381333441b0978242590eb0564b]). Expected version 1 got 0
I20240503 07:59:39.878788 6738 catalog_manager.cc:5576] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483) does not have the latest schema for tablet c9eef3b835ac41dfa3a2054128d7a14f (table rebalancer_dynamic_table [id=307cb381333441b0978242590eb0564b]). Expected version 1 got 0
I20240503 07:59:39.881701 18890 tablet.cc:1722] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:39.882166 18890 tablet.cc:1620] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:39.882272 18139 consensus_queue.cc:1035] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.886955 18892 tablet.cc:1722] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:39.887720 18892 tablet.cc:1620] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: MemRowSet was empty: no flush needed.
I20240503 07:59:39.895555 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=307cb381333441b0978242590eb0564b] alter complete (version 1)
I20240503 07:59:39.900875 17600 consensus_queue.cc:1035] T e33d88b13ce04b18876d4ce66b435d34 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.919131 9373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a86b58fdebf942ab870d4b354414b8d2"
dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35494
I20240503 07:59:39.919320 9373 raft_consensus.cc:595] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:39.983099 6747 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:39.990212 6868 tablet_service.cc:1467] Processing CreateTablet for tablet a1fa7fa545924c94b76ca281f805f4be (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=0c448f495c184df18f164145417dd190]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.990630 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fa7fa545924c94b76ca281f805f4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.990578 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 57657ba57e3947039ba34e8d3ad3e58f (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=0c448f495c184df18f164145417dd190]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.990638 7130 tablet_service.cc:1467] Processing CreateTablet for tablet a1fa7fa545924c94b76ca281f805f4be (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=0c448f495c184df18f164145417dd190]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.990708 9484 tablet_service.cc:1467] Processing CreateTablet for tablet ac7f37430cce4462a7ccf0996c251755 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=0c448f495c184df18f164145417dd190]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.990897 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fa7fa545924c94b76ca281f805f4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.990890 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57657ba57e3947039ba34e8d3ad3e58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.990973 9484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7f37430cce4462a7ccf0996c251755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.993273 18737 tablet_bootstrap.cc:492] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:39.993454 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 57657ba57e3947039ba34e8d3ad3e58f (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=0c448f495c184df18f164145417dd190]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.993739 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57657ba57e3947039ba34e8d3ad3e58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.994371 18737 tablet_bootstrap.cc:654] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.995137 18938 tablet_bootstrap.cc:492] T a1fa7fa545924c94b76ca281f805f4be P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:39.996233 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 57657ba57e3947039ba34e8d3ad3e58f (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=0c448f495c184df18f164145417dd190]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.996410 6868 tablet_service.cc:1467] Processing CreateTablet for tablet ac7f37430cce4462a7ccf0996c251755 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=0c448f495c184df18f164145417dd190]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.996518 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57657ba57e3947039ba34e8d3ad3e58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.996661 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7f37430cce4462a7ccf0996c251755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.996923 18938 tablet_bootstrap.cc:654] T a1fa7fa545924c94b76ca281f805f4be P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.998687 7261 tablet_service.cc:1467] Processing CreateTablet for tablet a1fa7fa545924c94b76ca281f805f4be (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=0c448f495c184df18f164145417dd190]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.998941 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fa7fa545924c94b76ca281f805f4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.999186 18737 tablet_bootstrap.cc:492] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a: No bootstrap required, opened a new log
I20240503 07:59:39.999272 18737 ts_tablet_manager.cc:1367] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 07:59:39.999775 18737 raft_consensus.cc:348] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:39.999890 18737 raft_consensus.cc:374] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.999912 18737 raft_consensus.cc:729] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.000020 18737 consensus_queue.cc:260] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:40.000322 18737 ts_tablet_manager.cc:1398] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:40.001370 6998 tablet_service.cc:1467] Processing CreateTablet for tablet ac7f37430cce4462a7ccf0996c251755 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=0c448f495c184df18f164145417dd190]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:40.001652 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7f37430cce4462a7ccf0996c251755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.003737 18736 tablet_bootstrap.cc:492] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:40.003808 18738 tablet_bootstrap.cc:492] T a1fa7fa545924c94b76ca281f805f4be P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:40.004745 18736 tablet_bootstrap.cc:654] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.004825 18738 tablet_bootstrap.cc:654] T a1fa7fa545924c94b76ca281f805f4be P 5227b26243484bbfb177a995ceb50afc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.005452 6746 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:40.006134 18736 tablet_bootstrap.cc:492] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d: No bootstrap required, opened a new log
I20240503 07:59:40.006222 18736 ts_tablet_manager.cc:1367] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:40.006721 18736 raft_consensus.cc:348] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:40.006827 18736 raft_consensus.cc:374] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.006855 18736 raft_consensus.cc:729] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.006961 18736 consensus_queue.cc:260] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:40.007225 18736 ts_tablet_manager.cc:1398] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:40.009224 18941 tablet_bootstrap.cc:492] T 57657ba57e3947039ba34e8d3ad3e58f P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:40.009526 18936 tablet_bootstrap.cc:492] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:40.011204 18937 tablet_bootstrap.cc:492] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:40.012312 18941 tablet_bootstrap.cc:654] T 57657ba57e3947039ba34e8d3ad3e58f P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.013083 18738 tablet_bootstrap.cc:492] T a1fa7fa545924c94b76ca281f805f4be P 5227b26243484bbfb177a995ceb50afc: No bootstrap required, opened a new log
I20240503 07:59:40.013185 18738 ts_tablet_manager.cc:1367] T a1fa7fa545924c94b76ca281f805f4be P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 07:59:40.013695 18738 raft_consensus.cc:348] T a1fa7fa545924c94b76ca281f805f4be P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:40.013823 18738 raft_consensus.cc:374] T a1fa7fa545924c94b76ca281f805f4be P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.013864 18738 raft_consensus.cc:729] T a1fa7fa545924c94b76ca281f805f4be P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.013828 18890 tablet.cc:1722] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:40.013993 18738 consensus_queue.cc:260] T a1fa7fa545924c94b76ca281f805f4be P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:40.014138 18890 tablet.cc:1620] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: MemRowSet was empty: no flush needed.
I20240503 07:59:40.014297 18937 tablet_bootstrap.cc:654] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.014323 18738 ts_tablet_manager.cc:1398] T a1fa7fa545924c94b76ca281f805f4be P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:40.016049 18892 tablet.cc:1722] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:40.016332 18936 tablet_bootstrap.cc:654] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.016368 18892 tablet.cc:1620] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: MemRowSet was empty: no flush needed.
I20240503 07:59:40.017655 18941 tablet_bootstrap.cc:492] T 57657ba57e3947039ba34e8d3ad3e58f P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:40.017834 18941 ts_tablet_manager.cc:1367] T 57657ba57e3947039ba34e8d3ad3e58f P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:40.018409 18941 raft_consensus.cc:348] T 57657ba57e3947039ba34e8d3ad3e58f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:40.018790 18941 raft_consensus.cc:374] T 57657ba57e3947039ba34e8d3ad3e58f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.018893 18941 raft_consensus.cc:729] T 57657ba57e3947039ba34e8d3ad3e58f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.019093 18941 consensus_queue.cc:260] T 57657ba57e3947039ba34e8d3ad3e58f P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:40.019474 18941 ts_tablet_manager.cc:1398] T 57657ba57e3947039ba34e8d3ad3e58f P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:40.021360 18936 tablet_bootstrap.cc:492] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:40.021509 18936 ts_tablet_manager.cc:1367] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240503 07:59:40.022128 18936 raft_consensus.cc:348] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:40.022398 18936 raft_consensus.cc:374] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.022485 18936 raft_consensus.cc:729] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.022681 18936 consensus_queue.cc:260] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:40.023059 18936 ts_tablet_manager.cc:1398] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:40.026465 18891 tablet.cc:1722] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:40.026834 18891 tablet.cc:1620] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: MemRowSet was empty: no flush needed.
I20240503 07:59:40.031317 6743 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table [id=307cb381333441b0978242590eb0564b] alter complete (version 2)
I20240503 07:59:40.051296 18548 raft_consensus.cc:482] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:40.051452 18548 raft_consensus.cc:504] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:40.051864 18548 leader_election.cc:288] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:40.052651 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7f37430cce4462a7ccf0996c251755" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:40.053088 18938 tablet_bootstrap.cc:492] T a1fa7fa545924c94b76ca281f805f4be P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:40.053234 18938 ts_tablet_manager.cc:1367] T a1fa7fa545924c94b76ca281f805f4be P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.000s
W20240503 07:59:40.053220 6953 leader_election.cc:341] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:40.053563 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7f37430cce4462a7ccf0996c251755" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:40.053712 9504 raft_consensus.cc:2454] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 0.
I20240503 07:59:40.053817 18938 raft_consensus.cc:348] T a1fa7fa545924c94b76ca281f805f4be P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:40.053959 18938 raft_consensus.cc:374] T a1fa7fa545924c94b76ca281f805f4be P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.054042 18938 raft_consensus.cc:729] T a1fa7fa545924c94b76ca281f805f4be P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.054102 6956 leader_election.cc:302] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters: c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:40.054188 18938 consensus_queue.cc:260] T a1fa7fa545924c94b76ca281f805f4be P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:40.054471 18548 raft_consensus.cc:2781] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:40.054538 18548 raft_consensus.cc:482] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:40.054580 18548 raft_consensus.cc:3037] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.054687 18938 ts_tablet_manager.cc:1398] T a1fa7fa545924c94b76ca281f805f4be P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.054886 18938 tablet_bootstrap.cc:492] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:40.055721 18548 raft_consensus.cc:504] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:40.056123 18548 leader_election.cc:288] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:40.056771 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7f37430cce4462a7ccf0996c251755" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:40.056833 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7f37430cce4462a7ccf0996c251755" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:40.056938 9504 raft_consensus.cc:3037] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:40.057142 6953 leader_election.cc:341] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:40.058040 9504 raft_consensus.cc:2454] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:40.058400 6956 leader_election.cc:302] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters: c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:40.059536 18938 tablet_bootstrap.cc:654] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.060658 18548 raft_consensus.cc:2781] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:40.060751 18548 raft_consensus.cc:686] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:40.060880 18548 consensus_queue.cc:237] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:40.061616 18937 tablet_bootstrap.cc:492] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:40.061743 18937 ts_tablet_manager.cc:1367] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.000s
I20240503 07:59:40.062269 18937 raft_consensus.cc:348] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:40.062431 18937 raft_consensus.cc:374] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.062462 18937 raft_consensus.cc:729] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.062593 18937 consensus_queue.cc:260] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:40.062924 18937 ts_tablet_manager.cc:1398] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.063180 18937 tablet_bootstrap.cc:492] T 57657ba57e3947039ba34e8d3ad3e58f P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:40.064214 18937 tablet_bootstrap.cc:654] T 57657ba57e3947039ba34e8d3ad3e58f P 00a3f8e159c64327832d79fb32cdcca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.064373 18264 raft_consensus.cc:979] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b: : Instructing follower 159177b3caeb434baa98e04136d050b2 to start an election
I20240503 07:59:40.064458 18264 raft_consensus.cc:1067] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Signalling peer 159177b3caeb434baa98e04136d050b2 to start an election
I20240503 07:59:40.065241 9635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a86b58fdebf942ab870d4b354414b8d2"
dest_uuid: "159177b3caeb434baa98e04136d050b2"
from {username='slave'} at 127.0.99.133:53879
I20240503 07:59:40.065410 9635 raft_consensus.cc:482] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:40.065466 9635 raft_consensus.cc:3037] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:40.065786 18937 tablet_bootstrap.cc:492] T 57657ba57e3947039ba34e8d3ad3e58f P 00a3f8e159c64327832d79fb32cdcca8: No bootstrap required, opened a new log
I20240503 07:59:40.065869 18937 ts_tablet_manager.cc:1367] T 57657ba57e3947039ba34e8d3ad3e58f P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:40.066516 18937 raft_consensus.cc:348] T 57657ba57e3947039ba34e8d3ad3e58f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:40.066620 18937 raft_consensus.cc:374] T 57657ba57e3947039ba34e8d3ad3e58f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.066651 18937 raft_consensus.cc:729] T 57657ba57e3947039ba34e8d3ad3e58f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.066766 18937 consensus_queue.cc:260] T 57657ba57e3947039ba34e8d3ad3e58f P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:40.066732 9635 raft_consensus.cc:504] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:40.067081 18937 ts_tablet_manager.cc:1398] T 57657ba57e3947039ba34e8d3ad3e58f P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.067246 9635 leader_election.cc:288] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:40.067518 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86b58fdebf942ab870d4b354414b8d2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:40.067642 6888 raft_consensus.cc:3037] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:40.067610 9373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86b58fdebf942ab870d4b354414b8d2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "3befbb2d206f4be89d3e07f38837374b"
I20240503 07:59:40.067741 9373 raft_consensus.cc:3032] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:40.067786 9373 raft_consensus.cc:729] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Running, Role: LEADER
I20240503 07:59:40.067940 9373 consensus_queue.cc:260] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:40.068111 9373 raft_consensus.cc:3037] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:40.068779 6888 raft_consensus.cc:2454] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 2.
I20240503 07:59:40.069027 9373 raft_consensus.cc:2454] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 2.
I20240503 07:59:40.069327 9767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86b58fdebf942ab870d4b354414b8d2" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:40.069485 9767 raft_consensus.cc:3037] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:40.070295 9767 raft_consensus.cc:2454] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 2.
I20240503 07:59:40.072607 9572 leader_election.cc:302] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:40.072803 18883 raft_consensus.cc:2781] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:40.072886 18883 raft_consensus.cc:686] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 2 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:40.073056 18883 consensus_queue.cc:237] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:40.077008 6747 catalog_manager.cc:5484] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:40.077366 6740 catalog_manager.cc:5484] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:40.087113 18938 tablet_bootstrap.cc:492] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:40.087255 18938 ts_tablet_manager.cc:1367] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240503 07:59:40.087815 18938 raft_consensus.cc:348] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:40.087957 18938 raft_consensus.cc:374] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.087990 18938 raft_consensus.cc:729] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.088128 18938 consensus_queue.cc:260] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:40.088443 18938 ts_tablet_manager.cc:1398] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.113266 17401 raft_consensus.cc:482] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:40.113446 17401 raft_consensus.cc:504] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:40.113919 17401 leader_election.cc:288] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:40.114493 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57657ba57e3947039ba34e8d3ad3e58f" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:40.114646 9635 raft_consensus.cc:2454] T 57657ba57e3947039ba34e8d3ad3e58f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:40.114964 9700 leader_election.cc:302] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:40.115128 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57657ba57e3947039ba34e8d3ad3e58f" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:40.115267 7150 raft_consensus.cc:2454] T 57657ba57e3947039ba34e8d3ad3e58f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 0.
I20240503 07:59:40.115494 17401 raft_consensus.cc:2781] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:40.115561 17401 raft_consensus.cc:482] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:40.115599 17401 raft_consensus.cc:3037] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.116483 17401 raft_consensus.cc:504] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:40.116837 17401 leader_election.cc:288] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:40.117116 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57657ba57e3947039ba34e8d3ad3e58f" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:40.117234 7150 raft_consensus.cc:3037] T 57657ba57e3947039ba34e8d3ad3e58f P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.117985 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57657ba57e3947039ba34e8d3ad3e58f" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2"
I20240503 07:59:40.118101 9635 raft_consensus.cc:3037] T 57657ba57e3947039ba34e8d3ad3e58f P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.118444 7150 raft_consensus.cc:2454] T 57657ba57e3947039ba34e8d3ad3e58f P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:40.119225 9635 raft_consensus.cc:2454] T 57657ba57e3947039ba34e8d3ad3e58f P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:40.119347 9703 leader_election.cc:302] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:40.119704 17401 raft_consensus.cc:2781] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:40.119789 17401 raft_consensus.cc:686] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:40.119927 17401 consensus_queue.cc:237] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:40.123010 6739 catalog_manager.cc:5484] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:40.132444 17804 raft_consensus.cc:482] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:40.132583 17804 raft_consensus.cc:504] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:40.133020 17804 leader_election.cc:288] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:40.133700 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fa7fa545924c94b76ca281f805f4be" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:40.133893 6888 raft_consensus.cc:2454] T a1fa7fa545924c94b76ca281f805f4be P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:40.134457 7084 leader_election.cc:302] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:40.134645 17804 raft_consensus.cc:2781] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:40.134712 17804 raft_consensus.cc:482] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:40.134778 17804 raft_consensus.cc:3037] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.136023 17804 raft_consensus.cc:504] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:40.136418 17804 leader_election.cc:288] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:40.138142 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fa7fa545924c94b76ca281f805f4be" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:40.138288 6888 raft_consensus.cc:3037] T a1fa7fa545924c94b76ca281f805f4be P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.139416 6888 raft_consensus.cc:2454] T a1fa7fa545924c94b76ca281f805f4be P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:40.144109 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:40.147042 7084 leader_election.cc:302] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:40.147562 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fa7fa545924c94b76ca281f805f4be" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:40.147660 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fa7fa545924c94b76ca281f805f4be" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:40.147711 7282 raft_consensus.cc:2454] T a1fa7fa545924c94b76ca281f805f4be P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 0.
I20240503 07:59:40.148998 17340 raft_consensus.cc:2781] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:40.149102 17340 raft_consensus.cc:686] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:40.149236 17340 consensus_queue.cc:237] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:40.153110 18890 tablet.cc:1722] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:40.153295 6739 catalog_manager.cc:5484] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:40.155334 18892 tablet.cc:1722] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:40.158855 18891 tablet.cc:1722] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:40.160123 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:40.162933 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:40.164072 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:40.169363 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:40.169524 9615 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 07:59:40.172250 6742 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: rebalancer_dynamic_table_new_name [id=307cb381333441b0978242590eb0564b] alter complete (version 3)
I20240503 07:59:40.177006 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:40.182503 9484 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:40.207130 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:40.242488 18696 raft_consensus.cc:482] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:40.242632 18696 raft_consensus.cc:504] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:40.243047 18696 leader_election.cc:288] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:40.243522 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7f37430cce4462a7ccf0996c251755" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:40.243692 9504 raft_consensus.cc:2379] T ac7f37430cce4462a7ccf0996c251755 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c4218904dd7e42d2be95b901a6e5db92 in current term 1: Already voted for candidate 9c085a3241f14cb7ba1332b508f4b57d in this term.
I20240503 07:59:40.243793 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7f37430cce4462a7ccf0996c251755" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:40.244491 6824 leader_election.cc:302] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92; no voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a
I20240503 07:59:40.245041 18696 raft_consensus.cc:3037] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.246309 18696 raft_consensus.cc:2726] T ac7f37430cce4462a7ccf0996c251755 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:40.288268 6739 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:40.288458 6739 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43696:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:40.291388 6739 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f72296bedf9648789ec7fcba5a38bea2: Sending DeleteTablet for 3 replicas of tablet c9eef3b835ac41dfa3a2054128d7a14f
I20240503 07:59:40.291916 9484 tablet_service.cc:1514] Processing DeleteTablet for tablet c9eef3b835ac41dfa3a2054128d7a14f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:40 UTC) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:40.292222 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet c9eef3b835ac41dfa3a2054128d7a14f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:40 UTC) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:40.292282 18995 tablet_replica.cc:330] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:40.292439 18995 raft_consensus.cc:2229] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:40.292711 18995 raft_consensus.cc:2258] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.293025 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet c9eef3b835ac41dfa3a2054128d7a14f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:40 UTC) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:40.293699 18995 ts_tablet_manager.cc:1875] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:40.294535 18997 tablet_replica.cc:330] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:40.294700 18997 raft_consensus.cc:2229] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:40.294831 18997 raft_consensus.cc:2258] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.295004 18996 tablet_replica.cc:330] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:40.295445 18997 ts_tablet_manager.cc:1875] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:40.296909 18995 ts_tablet_manager.cc:1888] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:40.296993 18995 log.cc:1196] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/c9eef3b835ac41dfa3a2054128d7a14f
I20240503 07:59:40.297237 18995 ts_tablet_manager.cc:1909] T c9eef3b835ac41dfa3a2054128d7a14f P cb73994515af4880b05383e78368bb4a: Deleting consensus metadata
I20240503 07:59:40.297590 18997 ts_tablet_manager.cc:1888] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:40.297741 18997 log.cc:1196] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c9eef3b835ac41dfa3a2054128d7a14f
I20240503 07:59:40.298061 18997 ts_tablet_manager.cc:1909] T c9eef3b835ac41dfa3a2054128d7a14f P 9c085a3241f14cb7ba1332b508f4b57d: Deleting consensus metadata
I20240503 07:59:40.298115 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet c9eef3b835ac41dfa3a2054128d7a14f (table rebalancer_dynamic_table_new_name [id=307cb381333441b0978242590eb0564b]) successfully deleted
I20240503 07:59:40.298498 18996 raft_consensus.cc:2229] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:40.298672 18996 raft_consensus.cc:2258] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.298743 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet c9eef3b835ac41dfa3a2054128d7a14f (table rebalancer_dynamic_table_new_name [id=307cb381333441b0978242590eb0564b]) successfully deleted
I20240503 07:59:40.299410 18996 ts_tablet_manager.cc:1875] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:40.301849 18996 ts_tablet_manager.cc:1888] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:40.301977 18996 log.cc:1196] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c9eef3b835ac41dfa3a2054128d7a14f
I20240503 07:59:40.302314 18996 ts_tablet_manager.cc:1909] T c9eef3b835ac41dfa3a2054128d7a14f P 5227b26243484bbfb177a995ceb50afc: Deleting consensus metadata
I20240503 07:59:40.303050 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet c9eef3b835ac41dfa3a2054128d7a14f (table rebalancer_dynamic_table_new_name [id=307cb381333441b0978242590eb0564b]) successfully deleted
I20240503 07:59:40.393955 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43696:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:40.401074 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 24db622340524fa7a973b84ad7766b45 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72f677cc55ac439c91e82e9d5fb6306d]), partition=
I20240503 07:59:40.401413 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24db622340524fa7a973b84ad7766b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.401420 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 24db622340524fa7a973b84ad7766b45 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72f677cc55ac439c91e82e9d5fb6306d]), partition=
I20240503 07:59:40.401664 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24db622340524fa7a973b84ad7766b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.402809 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 24db622340524fa7a973b84ad7766b45 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72f677cc55ac439c91e82e9d5fb6306d]), partition=
I20240503 07:59:40.403087 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24db622340524fa7a973b84ad7766b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.404892 18938 tablet_bootstrap.cc:492] T 24db622340524fa7a973b84ad7766b45 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:40.405069 18941 tablet_bootstrap.cc:492] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:40.405356 18936 tablet_bootstrap.cc:492] T 24db622340524fa7a973b84ad7766b45 P a372e98a670f49e9adca46ab5c836c4a: Bootstrap starting.
I20240503 07:59:40.405978 18938 tablet_bootstrap.cc:654] T 24db622340524fa7a973b84ad7766b45 P c4218904dd7e42d2be95b901a6e5db92: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.406034 18941 tablet_bootstrap.cc:654] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.406276 18936 tablet_bootstrap.cc:654] T 24db622340524fa7a973b84ad7766b45 P a372e98a670f49e9adca46ab5c836c4a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.407557 18938 tablet_bootstrap.cc:492] T 24db622340524fa7a973b84ad7766b45 P c4218904dd7e42d2be95b901a6e5db92: No bootstrap required, opened a new log
I20240503 07:59:40.407652 18938 ts_tablet_manager.cc:1367] T 24db622340524fa7a973b84ad7766b45 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:40.408206 18938 raft_consensus.cc:348] T 24db622340524fa7a973b84ad7766b45 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:40.408322 18938 raft_consensus.cc:374] T 24db622340524fa7a973b84ad7766b45 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.408357 18938 raft_consensus.cc:729] T 24db622340524fa7a973b84ad7766b45 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.408474 18938 consensus_queue.cc:260] T 24db622340524fa7a973b84ad7766b45 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:40.408788 18938 ts_tablet_manager.cc:1398] T 24db622340524fa7a973b84ad7766b45 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.411711 18936 tablet_bootstrap.cc:492] T 24db622340524fa7a973b84ad7766b45 P a372e98a670f49e9adca46ab5c836c4a: No bootstrap required, opened a new log
I20240503 07:59:40.411818 18936 ts_tablet_manager.cc:1367] T 24db622340524fa7a973b84ad7766b45 P a372e98a670f49e9adca46ab5c836c4a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 07:59:40.412431 18936 raft_consensus.cc:348] T 24db622340524fa7a973b84ad7766b45 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:40.412554 18936 raft_consensus.cc:374] T 24db622340524fa7a973b84ad7766b45 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.412580 18936 raft_consensus.cc:729] T 24db622340524fa7a973b84ad7766b45 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.412714 18936 consensus_queue.cc:260] T 24db622340524fa7a973b84ad7766b45 P a372e98a670f49e9adca46ab5c836c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:40.413028 18936 ts_tablet_manager.cc:1398] T 24db622340524fa7a973b84ad7766b45 P a372e98a670f49e9adca46ab5c836c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.427141 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:40.432569 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:40.438156 9615 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 07:59:40.438251 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:40.455686 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:40.457345 18941 tablet_bootstrap.cc:492] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2: No bootstrap required, opened a new log
I20240503 07:59:40.457485 18941 ts_tablet_manager.cc:1367] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.002s
I20240503 07:59:40.457990 18941 raft_consensus.cc:348] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:40.458110 18941 raft_consensus.cc:374] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.458135 18941 raft_consensus.cc:729] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.458261 18941 consensus_queue.cc:260] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:40.458799 18941 ts_tablet_manager.cc:1398] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.463554 9357 raft_consensus.cc:1261] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 2 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 07:59:40.464169 18513 consensus_queue.cc:1035] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:40.480538 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:40.491134 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:40.506768 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:40.561530 17401 consensus_queue.cc:1035] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.566684 18418 consensus_queue.cc:1035] T 57657ba57e3947039ba34e8d3ad3e58f P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.568725 9503 consensus_queue.cc:237] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:40.570273 9373 raft_consensus.cc:1261] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:40.570874 9766 raft_consensus.cc:1261] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:40.571353 18139 consensus_queue.cc:1035] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:40.571580 17600 consensus_queue.cc:1035] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:40.573511 17600 raft_consensus.cc:2932] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:40.574415 9766 raft_consensus.cc:2932] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:40.577093 9373 raft_consensus.cc:2932] T 5388c1fa25bc4f19a2aef9d8960e88aa P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:40.577872 6741 catalog_manager.cc:5484] T 5388c1fa25bc4f19a2aef9d8960e88aa P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index 2 to 3, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:40.581795 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet 5388c1fa25bc4f19a2aef9d8960e88aa with delete_type TABLET_DATA_TOMBSTONED (TS c4218904dd7e42d2be95b901a6e5db92 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:32810
I20240503 07:59:40.582556 19072 tablet_replica.cc:330] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92: stopping tablet replica
I20240503 07:59:40.582785 19072 raft_consensus.cc:2229] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:40.583017 19072 raft_consensus.cc:2258] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.583482 19072 ts_tablet_manager.cc:1875] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:40.586153 19072 ts_tablet_manager.cc:1888] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:40.586226 19072 log.cc:1196] T 5388c1fa25bc4f19a2aef9d8960e88aa P c4218904dd7e42d2be95b901a6e5db92: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5388c1fa25bc4f19a2aef9d8960e88aa
I20240503 07:59:40.586829 6731 catalog_manager.cc:4830] TS c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): tablet 5388c1fa25bc4f19a2aef9d8960e88aa (table rebalancer_extra_table_14 [id=f0fcbc958ed14af18b82b7bb244d2bfb]) successfully deleted
I20240503 07:59:40.600461 7282 raft_consensus.cc:3037] T a1fa7fa545924c94b76ca281f805f4be P 5227b26243484bbfb177a995ceb50afc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.605995 17804 consensus_queue.cc:1035] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.607466 17804 consensus_queue.cc:1035] T a1fa7fa545924c94b76ca281f805f4be P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.615022 18548 consensus_queue.cc:1035] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.621194 18143 consensus_queue.cc:1035] T ac7f37430cce4462a7ccf0996c251755 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.646461 9766 raft_consensus.cc:1261] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 07:59:40.648000 18513 consensus_queue.cc:1035] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:40.660977 6888 raft_consensus.cc:1261] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 07:59:40.662703 18883 consensus_queue.cc:1035] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:40.674237 19121 raft_consensus.cc:482] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:40.674510 19121 raft_consensus.cc:504] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:40.676242 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24db622340524fa7a973b84ad7766b45" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:40.676602 6889 raft_consensus.cc:2454] T 24db622340524fa7a973b84ad7766b45 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:40.676633 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24db622340524fa7a973b84ad7766b45" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:40.676774 9766 raft_consensus.cc:2454] T 24db622340524fa7a973b84ad7766b45 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 0.
I20240503 07:59:40.677163 9572 leader_election.cc:302] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:40.677408 19122 raft_consensus.cc:2781] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:40.677438 19121 leader_election.cc:288] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:40.677567 19122 raft_consensus.cc:482] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:40.677729 19122 raft_consensus.cc:3037] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.681142 19122 raft_consensus.cc:504] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:40.681644 19122 leader_election.cc:288] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:40.682166 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24db622340524fa7a973b84ad7766b45" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:40.682174 9766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24db622340524fa7a973b84ad7766b45" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:40.682309 6889 raft_consensus.cc:3037] T 24db622340524fa7a973b84ad7766b45 P c4218904dd7e42d2be95b901a6e5db92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.682308 9766 raft_consensus.cc:3037] T 24db622340524fa7a973b84ad7766b45 P a372e98a670f49e9adca46ab5c836c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.683301 6889 raft_consensus.cc:2454] T 24db622340524fa7a973b84ad7766b45 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:40.683492 9766 raft_consensus.cc:2454] T 24db622340524fa7a973b84ad7766b45 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:40.684435 9572 leader_election.cc:302] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:40.685230 19121 raft_consensus.cc:2781] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:40.685377 19121 raft_consensus.cc:686] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:40.685578 19121 consensus_queue.cc:237] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:40.688954 6743 catalog_manager.cc:5484] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:40.756911 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:40.760254 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:40.767719 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:40.775020 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:40.782068 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:40.782527 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:40.783141 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:40.790000 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:40.869401 7149 consensus_queue.cc:237] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:40.870847 9635 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2 [term 2 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 07:59:40.870873 9503 raft_consensus.cc:1261] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 07:59:40.871417 17804 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:40.871798 17804 consensus_queue.cc:1035] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:40.873564 17340 raft_consensus.cc:2932] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:40.874117 9635 raft_consensus.cc:2932] T 64d33ff4164b49a7948aa5afee40bf3f P 159177b3caeb434baa98e04136d050b2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:40.877019 9503 raft_consensus.cc:2932] T 64d33ff4164b49a7948aa5afee40bf3f P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:40.877250 6738 catalog_manager.cc:5484] T 64d33ff4164b49a7948aa5afee40bf3f P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index 6 to 8, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:40.881664 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d33ff4164b49a7948aa5afee40bf3f with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:40.882042 19172 tablet_replica.cc:330] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:40.882193 19172 raft_consensus.cc:2229] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:40.882333 19172 raft_consensus.cc:2258] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.883857 19172 ts_tablet_manager.cc:1875] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:40.886587 19172 ts_tablet_manager.cc:1888] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 07:59:40.886761 19172 log.cc:1196] T 64d33ff4164b49a7948aa5afee40bf3f P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/64d33ff4164b49a7948aa5afee40bf3f
I20240503 07:59:40.887403 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 64d33ff4164b49a7948aa5afee40bf3f (table rebalance_test_table_17 [id=d31d8fdb279949c181bcb09812091ff0]) successfully deleted
I20240503 07:59:40.912783 9373 consensus_queue.cc:237] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:40.914495 9503 raft_consensus.cc:1261] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.914616 6889 raft_consensus.cc:1261] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:40.914682 9310 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b -> Peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): Couldn't send request to peer 00a3f8e159c64327832d79fb32cdcca8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96289b8f77bb4241bade14b1011e1de4. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:40.914994 18264 consensus_queue.cc:1035] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:40.915510 18264 consensus_queue.cc:1035] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:40.918519 18524 raft_consensus.cc:2932] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:40.919800 6889 raft_consensus.cc:2932] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:40.922374 6740 catalog_manager.cc:5484] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:40.923410 9503 raft_consensus.cc:2932] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:40.956573 7150 consensus_queue.cc:237] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:40.958092 7279 raft_consensus.cc:1261] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:40.958267 9503 raft_consensus.cc:1261] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:40.958655 17340 consensus_queue.cc:1035] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 07:59:40.958900 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5365969434f142b8a5757a9a73bc109a. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:40.959020 17804 consensus_queue.cc:1035] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:40.962913 17804 raft_consensus.cc:2932] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:40.964839 7279 raft_consensus.cc:2932] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:40.965605 9503 raft_consensus.cc:2932] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:40.966312 6743 catalog_manager.cc:5484] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index 5 to 6, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:40.994707 9357 consensus_queue.cc:237] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:40.996630 6889 raft_consensus.cc:1261] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.996776 9635 raft_consensus.cc:1261] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.997278 18264 consensus_queue.cc:1035] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:40.997598 9310 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b -> Peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): Couldn't send request to peer 00a3f8e159c64327832d79fb32cdcca8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61b93171848746be89447d2ef90459a7. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:40.998175 18264 consensus_queue.cc:1035] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:41.000144 19196 raft_consensus.cc:2932] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:41.000581 6889 raft_consensus.cc:2932] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:41.004650 6738 catalog_manager.cc:5484] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:41.010030 9634 raft_consensus.cc:2932] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:41.150642 19122 consensus_queue.cc:1035] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:41.152987 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:41.155355 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:41.157634 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:41.160925 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:41.161149 18883 consensus_queue.cc:1035] T 24db622340524fa7a973b84ad7766b45 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:41.166404 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:41.179569 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:41.187402 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:41.189841 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:41.352160 19332 ts_tablet_manager.cc:902] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:41.354610 19332 tablet_copy_client.cc:308] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:41.355136 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 5365969434f142b8a5757a9a73bc109a from peer 3befbb2d206f4be89d3e07f38837374b ({username='slave'} at 127.0.99.133:52953)
I20240503 07:59:41.355271 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 5365969434f142b8a5757a9a73bc109a from peer 3befbb2d206f4be89d3e07f38837374b at {username='slave'} at 127.0.99.133:52953: session id = 3befbb2d206f4be89d3e07f38837374b-5365969434f142b8a5757a9a73bc109a
I20240503 07:59:41.356380 7171 tablet_copy_source_session.cc:205] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:41.358939 19332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5365969434f142b8a5757a9a73bc109a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:41.366147 19332 tablet_copy_client.cc:783] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:41.366406 19332 tablet_copy_client.cc:647] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:41.366554 19337 ts_tablet_manager.cc:902] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8: Initiating tablet copy from peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:41.367386 19337 tablet_copy_client.cc:308] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37815
I20240503 07:59:41.368006 19332 tablet_copy_client.cc:523] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:41.368634 9394 tablet_copy_service.cc:140] P 3befbb2d206f4be89d3e07f38837374b: Received BeginTabletCopySession request for tablet 96289b8f77bb4241bade14b1011e1de4 from peer 00a3f8e159c64327832d79fb32cdcca8 ({username='slave'} at 127.0.99.131:44395)
I20240503 07:59:41.369087 9394 tablet_copy_service.cc:161] P 3befbb2d206f4be89d3e07f38837374b: Beginning new tablet copy session on tablet 96289b8f77bb4241bade14b1011e1de4 from peer 00a3f8e159c64327832d79fb32cdcca8 at {username='slave'} at 127.0.99.131:44395: session id = 00a3f8e159c64327832d79fb32cdcca8-96289b8f77bb4241bade14b1011e1de4
I20240503 07:59:41.369647 19332 tablet_bootstrap.cc:492] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b: Bootstrap starting.
I20240503 07:59:41.370231 9394 tablet_copy_source_session.cc:205] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:41.377076 19337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96289b8f77bb4241bade14b1011e1de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:41.399137 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:41.401690 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:41.406905 19332 tablet_bootstrap.cc:492] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:41.407058 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:41.407311 19332 tablet_bootstrap.cc:492] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b: Bootstrap complete.
I20240503 07:59:41.407524 19332 ts_tablet_manager.cc:1367] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b: Time spent bootstrapping tablet: real 0.038s user 0.027s sys 0.000s
I20240503 07:59:41.408039 19332 raft_consensus.cc:348] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:41.408171 19332 raft_consensus.cc:729] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3befbb2d206f4be89d3e07f38837374b, State: Initialized, Role: FOLLOWER
I20240503 07:59:41.408295 19332 consensus_queue.cc:260] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:41.408572 19332 ts_tablet_manager.cc:1398] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:41.412104 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session 3befbb2d206f4be89d3e07f38837374b-5365969434f142b8a5757a9a73bc109a received from {username='slave'} at 127.0.99.133:52953
I20240503 07:59:41.412204 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session 3befbb2d206f4be89d3e07f38837374b-5365969434f142b8a5757a9a73bc109a on tablet 5365969434f142b8a5757a9a73bc109a with peer 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:41.439272 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:41.439559 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:41.439563 19354 ts_tablet_manager.cc:902] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8: Initiating tablet copy from peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:41.439626 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:41.441951 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:41.458554 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:41.462641 19354 tablet_copy_client.cc:308] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37815
I20240503 07:59:41.466189 9394 tablet_copy_service.cc:140] P 3befbb2d206f4be89d3e07f38837374b: Received BeginTabletCopySession request for tablet 61b93171848746be89447d2ef90459a7 from peer 00a3f8e159c64327832d79fb32cdcca8 ({username='slave'} at 127.0.99.131:44395)
I20240503 07:59:41.466331 9394 tablet_copy_service.cc:161] P 3befbb2d206f4be89d3e07f38837374b: Beginning new tablet copy session on tablet 61b93171848746be89447d2ef90459a7 from peer 00a3f8e159c64327832d79fb32cdcca8 at {username='slave'} at 127.0.99.131:44395: session id = 00a3f8e159c64327832d79fb32cdcca8-61b93171848746be89447d2ef90459a7
I20240503 07:59:41.467358 9394 tablet_copy_source_session.cc:205] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:41.469808 19354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b93171848746be89447d2ef90459a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:41.473184 19354 tablet_copy_client.cc:783] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:41.474993 19354 tablet_copy_client.cc:647] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:41.489732 19337 tablet_copy_client.cc:783] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:41.489935 19337 tablet_copy_client.cc:647] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:41.490162 19354 tablet_copy_client.cc:523] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:41.491209 19337 tablet_copy_client.cc:523] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:41.493050 19337 tablet_bootstrap.cc:492] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:41.493121 19354 tablet_bootstrap.cc:492] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:41.504259 19354 tablet_bootstrap.cc:492] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:41.504398 19337 tablet_bootstrap.cc:492] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:41.504561 19354 tablet_bootstrap.cc:492] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap complete.
I20240503 07:59:41.504650 19337 tablet_bootstrap.cc:492] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap complete.
I20240503 07:59:41.504746 19354 ts_tablet_manager.cc:1367] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240503 07:59:41.504906 19337 ts_tablet_manager.cc:1367] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240503 07:59:41.505373 19354 raft_consensus.cc:348] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:41.505412 19337 raft_consensus.cc:348] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:41.505501 19354 raft_consensus.cc:729] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:41.505518 19337 raft_consensus.cc:729] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:41.505622 19337 consensus_queue.cc:260] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:41.505667 19354 consensus_queue.cc:260] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:41.506285 19354 ts_tablet_manager.cc:1398] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.002s
I20240503 07:59:41.506906 9394 tablet_copy_service.cc:342] P 3befbb2d206f4be89d3e07f38837374b: Request end of tablet copy session 00a3f8e159c64327832d79fb32cdcca8-61b93171848746be89447d2ef90459a7 received from {username='slave'} at 127.0.99.131:44395
I20240503 07:59:41.507006 9394 tablet_copy_service.cc:433] P 3befbb2d206f4be89d3e07f38837374b: ending tablet copy session 00a3f8e159c64327832d79fb32cdcca8-61b93171848746be89447d2ef90459a7 on tablet 61b93171848746be89447d2ef90459a7 with peer 00a3f8e159c64327832d79fb32cdcca8
I20240503 07:59:41.508875 19337 ts_tablet_manager.cc:1398] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:41.509725 9394 tablet_copy_service.cc:342] P 3befbb2d206f4be89d3e07f38837374b: Request end of tablet copy session 00a3f8e159c64327832d79fb32cdcca8-96289b8f77bb4241bade14b1011e1de4 received from {username='slave'} at 127.0.99.131:44395
I20240503 07:59:41.509820 9394 tablet_copy_service.cc:433] P 3befbb2d206f4be89d3e07f38837374b: ending tablet copy session 00a3f8e159c64327832d79fb32cdcca8-96289b8f77bb4241bade14b1011e1de4 on tablet 96289b8f77bb4241bade14b1011e1de4 with peer 00a3f8e159c64327832d79fb32cdcca8
I20240503 07:59:41.554162 9503 consensus_queue.cc:237] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:41.555634 7279 raft_consensus.cc:1261] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:41.555815 6888 raft_consensus.cc:1261] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:41.556056 19362 consensus_queue.cc:1035] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:41.556222 17600 consensus_queue.cc:1035] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:41.558133 17600 raft_consensus.cc:2932] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:41.558514 7279 raft_consensus.cc:2932] T 287f2a8f05164e0eb722d53242e35023 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:41.558712 6888 raft_consensus.cc:2932] T 287f2a8f05164e0eb722d53242e35023 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:41.561990 6741 catalog_manager.cc:5484] T 287f2a8f05164e0eb722d53242e35023 P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:41.566103 9747 tablet_service.cc:1514] Processing DeleteTablet for tablet 287f2a8f05164e0eb722d53242e35023 with delete_type TABLET_DATA_TOMBSTONED (TS a372e98a670f49e9adca46ab5c836c4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:41.566937 19390 tablet_replica.cc:330] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:41.567111 19390 raft_consensus.cc:2229] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:41.567289 19390 raft_consensus.cc:2258] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:41.567862 19390 ts_tablet_manager.cc:1875] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:41.570410 19390 ts_tablet_manager.cc:1888] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:41.570500 19390 log.cc:1196] T 287f2a8f05164e0eb722d53242e35023 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/287f2a8f05164e0eb722d53242e35023
I20240503 07:59:41.571039 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 287f2a8f05164e0eb722d53242e35023 (table rebalancer_extra_table_26 [id=179f1c255c4d4e009e7f8119bc4e18be]) successfully deleted
I20240503 07:59:41.722872 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:41.723425 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:41.727739 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:41.732378 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:41.732440 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:41.735532 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:41.746551 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:41.751212 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:41.830736 9357 raft_consensus.cc:1203] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 07:59:41.843422 6886 consensus_queue.cc:237] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:41.844959 9634 raft_consensus.cc:1261] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:41.845260 7282 raft_consensus.cc:1261] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:41.845355 18551 consensus_queue.cc:1035] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:41.845739 18551 consensus_queue.cc:1035] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:41.847390 18551 raft_consensus.cc:2932] T 7bbaf13cbb474734a961664ae8ebe779 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:41.847811 9634 raft_consensus.cc:2932] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:41.849686 7282 raft_consensus.cc:2932] T 7bbaf13cbb474734a961664ae8ebe779 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:41.852339 6746 catalog_manager.cc:5484] T 7bbaf13cbb474734a961664ae8ebe779 P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:41.855824 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bbaf13cbb474734a961664ae8ebe779 with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:41.856528 19457 tablet_replica.cc:330] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:41.856731 19457 raft_consensus.cc:2229] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:41.856920 19457 raft_consensus.cc:2258] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:41.857699 19457 ts_tablet_manager.cc:1875] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:41.860502 19457 ts_tablet_manager.cc:1888] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:41.860666 19457 log.cc:1196] T 7bbaf13cbb474734a961664ae8ebe779 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7bbaf13cbb474734a961664ae8ebe779
I20240503 07:59:41.861346 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 7bbaf13cbb474734a961664ae8ebe779 (table rebalance_test_table_19 [id=071fb84151514ba094acc7ff3c1b24e7]) successfully deleted
I20240503 07:59:41.982870 7151 raft_consensus.cc:1203] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:42.037016 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:42.050755 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:42.051236 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:42.052254 7150 raft_consensus.cc:1203] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:42.053460 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:42.056540 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:42.056731 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:42.066202 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:42.116808 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:42.300781 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:42.302635 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:42.322683 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:42.323473 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:42.325753 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:42.328227 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:42.338577 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:42.347193 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:42.419435 7151 consensus_queue.cc:237] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:42.420804 9357 raft_consensus.cc:1261] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:42.420804 9505 raft_consensus.cc:1261] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:42.421454 19171 consensus_queue.cc:1035] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:42.421689 19213 consensus_queue.cc:1035] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:42.423668 19213 raft_consensus.cc:2932] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:42.424126 9357 raft_consensus.cc:2932] T 5365969434f142b8a5757a9a73bc109a P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:42.424369 9505 raft_consensus.cc:2932] T 5365969434f142b8a5757a9a73bc109a P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:42.427769 6744 catalog_manager.cc:5484] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index 6 to 7, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:42.449898 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet 5365969434f142b8a5757a9a73bc109a with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:42.450477 19618 tablet_replica.cc:330] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:42.450716 19618 raft_consensus.cc:2229] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:42.450975 19618 raft_consensus.cc:2258] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:42.452360 19618 ts_tablet_manager.cc:1875] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:42.461416 19618 ts_tablet_manager.cc:1888] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 07:59:42.461668 19618 log.cc:1196] T 5365969434f142b8a5757a9a73bc109a P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5365969434f142b8a5757a9a73bc109a
I20240503 07:59:42.462335 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 5365969434f142b8a5757a9a73bc109a (table rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]) successfully deleted
I20240503 07:59:42.622856 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:42.628854 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:42.633162 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:42.645476 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:42.650851 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:42.650945 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:42.651177 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:42.658496 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:42.754945 9635 consensus_queue.cc:237] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:42.756529 9767 raft_consensus.cc:1261] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:42.757381 18883 consensus_queue.cc:1035] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:42.758515 6888 raft_consensus.cc:1261] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:42.758988 19697 consensus_queue.cc:1035] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:42.759461 19121 raft_consensus.cc:2932] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:42.759804 9767 raft_consensus.cc:2932] T a86b58fdebf942ab870d4b354414b8d2 P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:42.761221 6888 raft_consensus.cc:2932] T a86b58fdebf942ab870d4b354414b8d2 P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:42.763684 6742 catalog_manager.cc:5484] T a86b58fdebf942ab870d4b354414b8d2 P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index 2 to 4, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
I20240503 07:59:42.767585 9351 tablet_service.cc:1514] Processing DeleteTablet for tablet a86b58fdebf942ab870d4b354414b8d2 with delete_type TABLET_DATA_TOMBSTONED (TS 3befbb2d206f4be89d3e07f38837374b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:42.768164 19721 tablet_replica.cc:330] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:42.768409 19721 raft_consensus.cc:2229] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:42.768537 19721 raft_consensus.cc:2258] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:42.769008 19721 ts_tablet_manager.cc:1875] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:42.771845 19721 ts_tablet_manager.cc:1888] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 07:59:42.772075 19721 log.cc:1196] T a86b58fdebf942ab870d4b354414b8d2 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/a86b58fdebf942ab870d4b354414b8d2
I20240503 07:59:42.772722 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet a86b58fdebf942ab870d4b354414b8d2 (table rebalancer_extra_table_20 [id=2f64f0c37c8a48f68cd6e052fc7420fb]) successfully deleted
I20240503 07:59:42.800482 7151 consensus_queue.cc:237] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:42.802104 7019 raft_consensus.cc:1261] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:42.802512 19213 consensus_queue.cc:1035] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:42.802696 9767 raft_consensus.cc:1261] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 07:59:42.803109 7087 consensus_peers.cc:485] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16c06ab5a3e54ed2910afa50069810cc. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:42.803298 19213 consensus_queue.cc:1035] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:42.805034 19744 raft_consensus.cc:2932] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:42.805521 9767 raft_consensus.cc:2932] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:42.807363 7019 raft_consensus.cc:2932] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:42.810654 6745 catalog_manager.cc:5484] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 5 to 6, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:42.831859 9634 consensus_queue.cc:237] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:42.833459 9357 raft_consensus.cc:1261] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:42.833459 9767 raft_consensus.cc:1261] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:42.833879 18883 consensus_queue.cc:1035] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:42.834120 9569 consensus_peers.cc:485] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 -> Peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Couldn't send request to peer c4218904dd7e42d2be95b901a6e5db92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d54f576fbe3d44ff8311505aef9855e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:42.834450 19121 consensus_queue.cc:1035] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:42.836643 18883 raft_consensus.cc:2932] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:42.837114 9357 raft_consensus.cc:2932] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:42.839756 9767 raft_consensus.cc:2932] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:42.840598 6746 catalog_manager.cc:5484] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:42.863925 7019 consensus_queue.cc:237] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:42.865492 9634 raft_consensus.cc:1261] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 07:59:42.865725 6956 consensus_peers.cc:485] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): Couldn't send request to peer 00a3f8e159c64327832d79fb32cdcca8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:42.865764 7282 raft_consensus.cc:1261] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:42.865892 19782 consensus_queue.cc:1035] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:42.866153 19514 consensus_queue.cc:1035] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:42.868263 19698 raft_consensus.cc:2932] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:42.868693 7282 raft_consensus.cc:2932] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:42.871173 9634 raft_consensus.cc:2932] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:42.872031 6744 catalog_manager.cc:5484] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index 5 to 6, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:42.895476 9357 consensus_queue.cc:237] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
W20240503 07:59:42.897150 9310 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 659657d85679433abdcbe711e4741b67. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:42.897153 9767 raft_consensus.cc:1261] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:42.897415 9634 raft_consensus.cc:1261] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:42.897574 19801 consensus_queue.cc:1035] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:42.897984 18264 consensus_queue.cc:1035] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:42.900138 18264 raft_consensus.cc:2932] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:42.900851 9767 raft_consensus.cc:2932] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:42.902521 9634 raft_consensus.cc:2932] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:42.905395 6745 catalog_manager.cc:5484] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index -1 to 2, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:42.929334 9767 consensus_queue.cc:237] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:42.931015 9504 raft_consensus.cc:1261] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:42.931366 9634 raft_consensus.cc:1261] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:42.931715 9703 consensus_peers.cc:485] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a -> Peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): Couldn't send request to peer 00a3f8e159c64327832d79fb32cdcca8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf3d2c6d70bb49fcb1dd23fc0d402796. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:42.931768 19527 consensus_queue.cc:1035] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:42.932134 19528 consensus_queue.cc:1035] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:42.937450 19528 raft_consensus.cc:2932] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:42.937796 9634 raft_consensus.cc:2932] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:42.939831 9504 raft_consensus.cc:2932] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:42.941560 6746 catalog_manager.cc:5484] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:42.975365 7151 consensus_queue.cc:237] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:42.976974 9357 raft_consensus.cc:1261] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:42.977078 7086 consensus_peers.cc:485] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): Couldn't send request to peer 9c085a3241f14cb7ba1332b508f4b57d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b607148b2a7b4b56b83e9606be49221b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:42.977357 19171 consensus_queue.cc:1035] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:42.982892 9634 raft_consensus.cc:1261] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:42.983325 19171 consensus_queue.cc:1035] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:42.985302 19213 raft_consensus.cc:2932] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:42.985754 9634 raft_consensus.cc:2932] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:42.988471 9357 raft_consensus.cc:2932] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:42.989245 6745 catalog_manager.cc:5484] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:43.028585 9634 consensus_queue.cc:237] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:43.030226 9357 raft_consensus.cc:1261] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:43.030632 18883 consensus_queue.cc:1035] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:43.030915 9569 consensus_peers.cc:485] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 -> Peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655): Couldn't send request to peer c4218904dd7e42d2be95b901a6e5db92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96edc0386e0e45a4b9df0f47820c723c. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:43.031064 9505 raft_consensus.cc:1261] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:43.031432 18883 consensus_queue.cc:1035] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:43.033252 19765 raft_consensus.cc:2932] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:43.033735 9505 raft_consensus.cc:2932] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:43.036142 9357 raft_consensus.cc:2932] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:43.038300 6742 catalog_manager.cc:5484] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index -1 to 2, VOTER c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:43.060168 6888 consensus_queue.cc:237] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:43.061612 9634 raft_consensus.cc:1261] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:43.061744 9766 raft_consensus.cc:1261] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:43.062070 19877 consensus_queue.cc:1035] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:43.062276 18551 consensus_queue.cc:1035] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:43.062496 6825 consensus_peers.cc:485] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): Couldn't send request to peer 5227b26243484bbfb177a995ceb50afc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3503f0e1e29430a933986978978f1d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:43.064469 19877 raft_consensus.cc:2932] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:43.065049 9634 raft_consensus.cc:2932] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:43.068298 6746 catalog_manager.cc:5484] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:43.070202 9766 raft_consensus.cc:2932] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:43.089926 7151 consensus_queue.cc:237] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:43.091451 7019 raft_consensus.cc:1261] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:43.091501 7282 raft_consensus.cc:1261] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:43.091604 7087 consensus_peers.cc:485] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): Couldn't send request to peer cb73994515af4880b05383e78368bb4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 660355a4482d4a30bbb71cd0b56527d1. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:43.091886 19213 consensus_queue.cc:1035] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:43.092077 19171 consensus_queue.cc:1035] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:43.094283 19213 raft_consensus.cc:2932] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:43.094756 7019 raft_consensus.cc:2932] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:43.096756 7282 raft_consensus.cc:2932] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:43.099582 6745 catalog_manager.cc:5484] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: config changed from index -1 to 4, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:43.121673 9504 consensus_queue.cc:237] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
W20240503 07:59:43.123667 9441 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): Couldn't send request to peer 00a3f8e159c64327832d79fb32cdcca8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9cfe98aefcfe4380bb64f14df1a24318. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:43.124012 9357 raft_consensus.cc:1261] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:43.124326 9634 raft_consensus.cc:1261] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:43.124698 19914 consensus_queue.cc:1035] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:43.124897 17600 consensus_queue.cc:1035] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:43.127135 19914 raft_consensus.cc:2932] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:43.127447 9634 raft_consensus.cc:2932] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:43.130973 6742 catalog_manager.cc:5484] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:43.133332 9357 raft_consensus.cc:2932] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:43.241410 19959 ts_tablet_manager.cc:902] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92: Initiating tablet copy from peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:43.242084 19959 tablet_copy_client.cc:308] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:43369
I20240503 07:59:43.242697 9656 tablet_copy_service.cc:140] P 159177b3caeb434baa98e04136d050b2: Received BeginTabletCopySession request for tablet d54f576fbe3d44ff8311505aef9855e2 from peer c4218904dd7e42d2be95b901a6e5db92 ({username='slave'} at 127.0.99.129:40095)
I20240503 07:59:43.242815 9656 tablet_copy_service.cc:161] P 159177b3caeb434baa98e04136d050b2: Beginning new tablet copy session on tablet d54f576fbe3d44ff8311505aef9855e2 from peer c4218904dd7e42d2be95b901a6e5db92 at {username='slave'} at 127.0.99.129:40095: session id = c4218904dd7e42d2be95b901a6e5db92-d54f576fbe3d44ff8311505aef9855e2
I20240503 07:59:43.243860 9656 tablet_copy_source_session.cc:205] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:43.247069 19959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d54f576fbe3d44ff8311505aef9855e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:43.252601 19959 tablet_copy_client.cc:783] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:43.253222 19959 tablet_copy_client.cc:647] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:43.256918 19959 tablet_copy_client.cc:523] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:43.258376 19959 tablet_bootstrap.cc:492] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:43.264523 19964 ts_tablet_manager.cc:902] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Initiating tablet copy from peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:43.265269 19964 tablet_copy_client.cc:308] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46383
I20240503 07:59:43.266415 7039 tablet_copy_service.cc:140] P 9c085a3241f14cb7ba1332b508f4b57d: Received BeginTabletCopySession request for tablet 31c76213f3ec451084ae7b0551cd7b48 from peer 00a3f8e159c64327832d79fb32cdcca8 ({username='slave'} at 127.0.99.131:48979)
I20240503 07:59:43.266541 7039 tablet_copy_service.cc:161] P 9c085a3241f14cb7ba1332b508f4b57d: Beginning new tablet copy session on tablet 31c76213f3ec451084ae7b0551cd7b48 from peer 00a3f8e159c64327832d79fb32cdcca8 at {username='slave'} at 127.0.99.131:48979: session id = 00a3f8e159c64327832d79fb32cdcca8-31c76213f3ec451084ae7b0551cd7b48
I20240503 07:59:43.267472 7039 tablet_copy_source_session.cc:205] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:43.268266 19964 ts_tablet_manager.cc:1875] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 07:59:43.269055 19959 tablet_bootstrap.cc:492] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:43.269414 19959 tablet_bootstrap.cc:492] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92: Bootstrap complete.
I20240503 07:59:43.269940 19959 ts_tablet_manager.cc:1367] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240503 07:59:43.271412 19964 ts_tablet_manager.cc:1888] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 07:59:43.271793 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c76213f3ec451084ae7b0551cd7b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:43.271230 19959 raft_consensus.cc:348] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:43.274931 19959 raft_consensus.cc:729] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:43.275103 19959 consensus_queue.cc:260] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:43.279378 19959 ts_tablet_manager.cc:1398] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:43.281816 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:43.281661 19964 tablet_copy_client.cc:783] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:43.282130 19964 tablet_copy_client.cc:647] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:43.282718 9656 tablet_copy_service.cc:342] P 159177b3caeb434baa98e04136d050b2: Request end of tablet copy session c4218904dd7e42d2be95b901a6e5db92-d54f576fbe3d44ff8311505aef9855e2 received from {username='slave'} at 127.0.99.129:40095
I20240503 07:59:43.282801 9656 tablet_copy_service.cc:433] P 159177b3caeb434baa98e04136d050b2: ending tablet copy session c4218904dd7e42d2be95b901a6e5db92-d54f576fbe3d44ff8311505aef9855e2 on tablet d54f576fbe3d44ff8311505aef9855e2 with peer c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:43.285835 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:43.294071 19964 tablet_copy_client.cc:523] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:43.296495 19964 tablet_bootstrap.cc:492] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:43.300748 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:43.302898 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:43.303393 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:43.319525 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:43.331717 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:43.332121 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:43.363737 19964 tablet_bootstrap.cc:492] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:43.364102 19964 tablet_bootstrap.cc:492] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap complete.
I20240503 07:59:43.364284 19964 ts_tablet_manager.cc:1367] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.068s user 0.026s sys 0.000s
I20240503 07:59:43.364748 19964 raft_consensus.cc:348] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:43.364878 19964 raft_consensus.cc:729] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:43.365090 19964 consensus_queue.cc:260] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:43.365378 19964 ts_tablet_manager.cc:1398] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:59:43.366437 7039 tablet_copy_service.cc:342] P 9c085a3241f14cb7ba1332b508f4b57d: Request end of tablet copy session 00a3f8e159c64327832d79fb32cdcca8-31c76213f3ec451084ae7b0551cd7b48 received from {username='slave'} at 127.0.99.131:48979
I20240503 07:59:43.366521 7039 tablet_copy_service.cc:433] P 9c085a3241f14cb7ba1332b508f4b57d: ending tablet copy session 00a3f8e159c64327832d79fb32cdcca8-31c76213f3ec451084ae7b0551cd7b48 on tablet 31c76213f3ec451084ae7b0551cd7b48 with peer 00a3f8e159c64327832d79fb32cdcca8
I20240503 07:59:43.390580 19983 ts_tablet_manager.cc:902] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:43.391160 19983 tablet_copy_client.cc:308] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37815
I20240503 07:59:43.391826 9394 tablet_copy_service.cc:140] P 3befbb2d206f4be89d3e07f38837374b: Received BeginTabletCopySession request for tablet 659657d85679433abdcbe711e4741b67 from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:44093)
I20240503 07:59:43.391947 9394 tablet_copy_service.cc:161] P 3befbb2d206f4be89d3e07f38837374b: Beginning new tablet copy session on tablet 659657d85679433abdcbe711e4741b67 from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:44093: session id = cb73994515af4880b05383e78368bb4a-659657d85679433abdcbe711e4741b67
I20240503 07:59:43.392922 9394 tablet_copy_source_session.cc:205] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:43.395895 19983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659657d85679433abdcbe711e4741b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:43.398334 19983 tablet_copy_client.cc:783] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:43.398538 19983 tablet_copy_client.cc:647] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:43.400358 19983 tablet_copy_client.cc:523] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:43.403630 19983 tablet_bootstrap.cc:492] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:43.413928 20000 ts_tablet_manager.cc:902] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:43.414774 20000 tablet_copy_client.cc:308] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:43.415457 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 16c06ab5a3e54ed2910afa50069810cc from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:48617)
I20240503 07:59:43.415581 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 16c06ab5a3e54ed2910afa50069810cc from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:48617: session id = cb73994515af4880b05383e78368bb4a-16c06ab5a3e54ed2910afa50069810cc
I20240503 07:59:43.416337 7171 tablet_copy_source_session.cc:205] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:43.417254 20000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c06ab5a3e54ed2910afa50069810cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:43.419734 19983 tablet_bootstrap.cc:492] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:43.420001 19983 tablet_bootstrap.cc:492] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:43.420174 19983 ts_tablet_manager.cc:1367] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20240503 07:59:43.420974 19983 raft_consensus.cc:348] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:43.421110 19983 raft_consensus.cc:729] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:43.421237 19983 consensus_queue.cc:260] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:43.421581 19983 ts_tablet_manager.cc:1398] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:43.422044 20000 tablet_copy_client.cc:783] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:43.422214 20000 tablet_copy_client.cc:647] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:43.422223 9394 tablet_copy_service.cc:342] P 3befbb2d206f4be89d3e07f38837374b: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-659657d85679433abdcbe711e4741b67 received from {username='slave'} at 127.0.99.134:44093
I20240503 07:59:43.422317 9394 tablet_copy_service.cc:433] P 3befbb2d206f4be89d3e07f38837374b: ending tablet copy session cb73994515af4880b05383e78368bb4a-659657d85679433abdcbe711e4741b67 on tablet 659657d85679433abdcbe711e4741b67 with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:43.428066 20000 tablet_copy_client.cc:523] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:43.429653 20000 tablet_bootstrap.cc:492] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:43.437630 19964 ts_tablet_manager.cc:902] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8: Initiating tablet copy from peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:43.438189 19964 tablet_copy_client.cc:308] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:43683
I20240503 07:59:43.438741 9787 tablet_copy_service.cc:140] P a372e98a670f49e9adca46ab5c836c4a: Received BeginTabletCopySession request for tablet cf3d2c6d70bb49fcb1dd23fc0d402796 from peer 00a3f8e159c64327832d79fb32cdcca8 ({username='slave'} at 127.0.99.131:49907)
I20240503 07:59:43.438865 9787 tablet_copy_service.cc:161] P a372e98a670f49e9adca46ab5c836c4a: Beginning new tablet copy session on tablet cf3d2c6d70bb49fcb1dd23fc0d402796 from peer 00a3f8e159c64327832d79fb32cdcca8 at {username='slave'} at 127.0.99.131:49907: session id = 00a3f8e159c64327832d79fb32cdcca8-cf3d2c6d70bb49fcb1dd23fc0d402796
I20240503 07:59:43.439957 9787 tablet_copy_source_session.cc:205] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:43.441587 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3d2c6d70bb49fcb1dd23fc0d402796. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:43.446190 19964 tablet_copy_client.cc:783] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:43.446586 19964 tablet_copy_client.cc:647] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:43.447927 19964 tablet_copy_client.cc:523] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:43.449589 19964 tablet_bootstrap.cc:492] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:43.453581 19964 tablet_bootstrap.cc:492] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:43.454067 19964 tablet_bootstrap.cc:492] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap complete.
I20240503 07:59:43.454315 19964 ts_tablet_manager.cc:1367] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:43.454931 19964 raft_consensus.cc:348] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:43.455153 19964 raft_consensus.cc:729] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:43.455375 19964 consensus_queue.cc:260] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:43.455847 19964 ts_tablet_manager.cc:1398] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:43.456491 9787 tablet_copy_service.cc:342] P a372e98a670f49e9adca46ab5c836c4a: Request end of tablet copy session 00a3f8e159c64327832d79fb32cdcca8-cf3d2c6d70bb49fcb1dd23fc0d402796 received from {username='slave'} at 127.0.99.131:49907
I20240503 07:59:43.456579 9787 tablet_copy_service.cc:433] P a372e98a670f49e9adca46ab5c836c4a: ending tablet copy session 00a3f8e159c64327832d79fb32cdcca8-cf3d2c6d70bb49fcb1dd23fc0d402796 on tablet cf3d2c6d70bb49fcb1dd23fc0d402796 with peer 00a3f8e159c64327832d79fb32cdcca8
I20240503 07:59:43.461880 20000 tablet_bootstrap.cc:492] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:43.462200 20000 tablet_bootstrap.cc:492] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:43.462435 20000 ts_tablet_manager.cc:1367] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.002s
I20240503 07:59:43.462965 20000 raft_consensus.cc:348] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:43.463101 20000 raft_consensus.cc:729] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:43.463331 20000 consensus_queue.cc:260] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:43.463763 20000 ts_tablet_manager.cc:1398] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:43.464576 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-16c06ab5a3e54ed2910afa50069810cc received from {username='slave'} at 127.0.99.134:48617
I20240503 07:59:43.464668 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session cb73994515af4880b05383e78368bb4a-16c06ab5a3e54ed2910afa50069810cc on tablet 16c06ab5a3e54ed2910afa50069810cc with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:43.501384 20041 ts_tablet_manager.cc:902] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:43.502077 20041 tablet_copy_client.cc:308] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:43.502553 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet b607148b2a7b4b56b83e9606be49221b from peer 9c085a3241f14cb7ba1332b508f4b57d ({username='slave'} at 127.0.99.130:45195)
I20240503 07:59:43.502669 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet b607148b2a7b4b56b83e9606be49221b from peer 9c085a3241f14cb7ba1332b508f4b57d at {username='slave'} at 127.0.99.130:45195: session id = 9c085a3241f14cb7ba1332b508f4b57d-b607148b2a7b4b56b83e9606be49221b
I20240503 07:59:43.503551 7171 tablet_copy_source_session.cc:205] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:43.504609 20041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b607148b2a7b4b56b83e9606be49221b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:43.507721 20041 tablet_copy_client.cc:783] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:43.508596 20041 tablet_copy_client.cc:647] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:43.510000 20041 tablet_copy_client.cc:523] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:43.512140 20041 tablet_bootstrap.cc:492] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:43.516513 20041 tablet_bootstrap.cc:492] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:43.516779 20041 tablet_bootstrap.cc:492] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap complete.
I20240503 07:59:43.516990 20041 ts_tablet_manager.cc:1367] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:43.517514 20041 raft_consensus.cc:348] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:43.517661 20041 raft_consensus.cc:729] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:43.517830 20041 consensus_queue.cc:260] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:43.518258 20041 ts_tablet_manager.cc:1398] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240503 07:59:43.518754 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-b607148b2a7b4b56b83e9606be49221b received from {username='slave'} at 127.0.99.130:45195
I20240503 07:59:43.518839 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-b607148b2a7b4b56b83e9606be49221b on tablet b607148b2a7b4b56b83e9606be49221b with peer 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:43.519963 19964 ts_tablet_manager.cc:902] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8: Initiating tablet copy from peer cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:43.520459 19964 tablet_copy_client.cc:308] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:45435
I20240503 07:59:43.520937 9525 tablet_copy_service.cc:140] P cb73994515af4880b05383e78368bb4a: Received BeginTabletCopySession request for tablet 9cfe98aefcfe4380bb64f14df1a24318 from peer 00a3f8e159c64327832d79fb32cdcca8 ({username='slave'} at 127.0.99.131:33047)
I20240503 07:59:43.521059 9525 tablet_copy_service.cc:161] P cb73994515af4880b05383e78368bb4a: Beginning new tablet copy session on tablet 9cfe98aefcfe4380bb64f14df1a24318 from peer 00a3f8e159c64327832d79fb32cdcca8 at {username='slave'} at 127.0.99.131:33047: session id = 00a3f8e159c64327832d79fb32cdcca8-9cfe98aefcfe4380bb64f14df1a24318
I20240503 07:59:43.522133 9525 tablet_copy_source_session.cc:205] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:43.523222 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfe98aefcfe4380bb64f14df1a24318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:43.525600 19964 tablet_copy_client.cc:783] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:43.525765 19964 tablet_copy_client.cc:647] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:43.530990 19964 tablet_copy_client.cc:523] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:43.532466 19964 tablet_bootstrap.cc:492] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
I20240503 07:59:43.539812 19964 tablet_bootstrap.cc:492] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:43.540170 19964 tablet_bootstrap.cc:492] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap complete.
I20240503 07:59:43.540370 19964 ts_tablet_manager.cc:1367] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 07:59:43.541325 19964 raft_consensus.cc:348] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:43.541461 19964 raft_consensus.cc:729] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:43.541626 19964 consensus_queue.cc:260] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:43.541951 19964 ts_tablet_manager.cc:1398] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:43.543071 9525 tablet_copy_service.cc:342] P cb73994515af4880b05383e78368bb4a: Request end of tablet copy session 00a3f8e159c64327832d79fb32cdcca8-9cfe98aefcfe4380bb64f14df1a24318 received from {username='slave'} at 127.0.99.131:33047
I20240503 07:59:43.543164 9525 tablet_copy_service.cc:433] P cb73994515af4880b05383e78368bb4a: ending tablet copy session 00a3f8e159c64327832d79fb32cdcca8-9cfe98aefcfe4380bb64f14df1a24318 on tablet 9cfe98aefcfe4380bb64f14df1a24318 with peer 00a3f8e159c64327832d79fb32cdcca8
I20240503 07:59:43.544215 19959 ts_tablet_manager.cc:902] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92: Initiating tablet copy from peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:43.544667 19959 tablet_copy_client.cc:308] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:43369
I20240503 07:59:43.545766 9656 tablet_copy_service.cc:140] P 159177b3caeb434baa98e04136d050b2: Received BeginTabletCopySession request for tablet 96edc0386e0e45a4b9df0f47820c723c from peer c4218904dd7e42d2be95b901a6e5db92 ({username='slave'} at 127.0.99.129:40095)
I20240503 07:59:43.545881 9656 tablet_copy_service.cc:161] P 159177b3caeb434baa98e04136d050b2: Beginning new tablet copy session on tablet 96edc0386e0e45a4b9df0f47820c723c from peer c4218904dd7e42d2be95b901a6e5db92 at {username='slave'} at 127.0.99.129:40095: session id = c4218904dd7e42d2be95b901a6e5db92-96edc0386e0e45a4b9df0f47820c723c
I20240503 07:59:43.546980 9656 tablet_copy_source_session.cc:205] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:43.548149 19959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96edc0386e0e45a4b9df0f47820c723c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:43.551352 19959 tablet_copy_client.cc:783] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:43.552031 19959 tablet_copy_client.cc:647] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:43.557860 19959 tablet_copy_client.cc:523] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:43.559688 19959 tablet_bootstrap.cc:492] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92: Bootstrap starting.
I20240503 07:59:43.564512 19959 tablet_bootstrap.cc:492] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:43.565515 19959 tablet_bootstrap.cc:492] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92: Bootstrap complete.
I20240503 07:59:43.565726 19959 ts_tablet_manager.cc:1367] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240503 07:59:43.566249 19959 raft_consensus.cc:348] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:43.566550 19959 raft_consensus.cc:729] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Initialized, Role: FOLLOWER
I20240503 07:59:43.566709 19959 consensus_queue.cc:260] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:43.567083 19959 ts_tablet_manager.cc:1398] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:43.568324 9656 tablet_copy_service.cc:342] P 159177b3caeb434baa98e04136d050b2: Request end of tablet copy session c4218904dd7e42d2be95b901a6e5db92-96edc0386e0e45a4b9df0f47820c723c received from {username='slave'} at 127.0.99.129:40095
I20240503 07:59:43.568411 9656 tablet_copy_service.cc:433] P 159177b3caeb434baa98e04136d050b2: ending tablet copy session c4218904dd7e42d2be95b901a6e5db92-96edc0386e0e45a4b9df0f47820c723c on tablet 96edc0386e0e45a4b9df0f47820c723c with peer c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:43.579177 20000 ts_tablet_manager.cc:902] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a: Initiating tablet copy from peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:43.579663 20000 tablet_copy_client.cc:308] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36317
I20240503 07:59:43.580158 7171 tablet_copy_service.cc:140] P 00a3f8e159c64327832d79fb32cdcca8: Received BeginTabletCopySession request for tablet 660355a4482d4a30bbb71cd0b56527d1 from peer cb73994515af4880b05383e78368bb4a ({username='slave'} at 127.0.99.134:48617)
I20240503 07:59:43.580287 7171 tablet_copy_service.cc:161] P 00a3f8e159c64327832d79fb32cdcca8: Beginning new tablet copy session on tablet 660355a4482d4a30bbb71cd0b56527d1 from peer cb73994515af4880b05383e78368bb4a at {username='slave'} at 127.0.99.134:48617: session id = cb73994515af4880b05383e78368bb4a-660355a4482d4a30bbb71cd0b56527d1
I20240503 07:59:43.581404 7171 tablet_copy_source_session.cc:205] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:43.582824 20000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660355a4482d4a30bbb71cd0b56527d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:43.585422 20000 tablet_copy_client.cc:783] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:43.585594 20000 tablet_copy_client.cc:647] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:43.587065 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:43.591953 20000 tablet_copy_client.cc:523] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:43.594527 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:43.596370 20000 tablet_bootstrap.cc:492] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a: Bootstrap starting.
I20240503 07:59:43.599751 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:43.612113 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:43.615366 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:43.619207 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:43.624110 20072 ts_tablet_manager.cc:902] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:43.627646 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:43.633248 20000 tablet_bootstrap.cc:492] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:43.633646 20000 tablet_bootstrap.cc:492] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a: Bootstrap complete.
I20240503 07:59:43.633863 20000 ts_tablet_manager.cc:1367] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a: Time spent bootstrapping tablet: real 0.038s user 0.023s sys 0.000s
I20240503 07:59:43.634526 20000 raft_consensus.cc:348] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:43.634685 20000 raft_consensus.cc:729] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb73994515af4880b05383e78368bb4a, State: Initialized, Role: FOLLOWER
I20240503 07:59:43.634850 20000 consensus_queue.cc:260] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:43.635171 20000 ts_tablet_manager.cc:1398] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:43.637073 7171 tablet_copy_service.cc:342] P 00a3f8e159c64327832d79fb32cdcca8: Request end of tablet copy session cb73994515af4880b05383e78368bb4a-660355a4482d4a30bbb71cd0b56527d1 received from {username='slave'} at 127.0.99.134:48617
I20240503 07:59:43.637171 7171 tablet_copy_service.cc:433] P 00a3f8e159c64327832d79fb32cdcca8: ending tablet copy session cb73994515af4880b05383e78368bb4a-660355a4482d4a30bbb71cd0b56527d1 on tablet 660355a4482d4a30bbb71cd0b56527d1 with peer cb73994515af4880b05383e78368bb4a
I20240503 07:59:43.638779 20072 tablet_copy_client.cc:308] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:43.639394 6908 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet b3503f0e1e29430a933986978978f1d4 from peer 5227b26243484bbfb177a995ceb50afc ({username='slave'} at 127.0.99.132:55149)
I20240503 07:59:43.639508 6908 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet b3503f0e1e29430a933986978978f1d4 from peer 5227b26243484bbfb177a995ceb50afc at {username='slave'} at 127.0.99.132:55149: session id = 5227b26243484bbfb177a995ceb50afc-b3503f0e1e29430a933986978978f1d4
I20240503 07:59:43.640758 6908 tablet_copy_source_session.cc:205] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:43.647279 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:43.648725 20072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3503f0e1e29430a933986978978f1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:43.653858 20072 tablet_copy_client.cc:783] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:43.654147 20072 tablet_copy_client.cc:647] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:43.674408 20072 tablet_copy_client.cc:523] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:43.677594 20072 tablet_bootstrap.cc:492] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:43.681650 20072 tablet_bootstrap.cc:492] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:43.681908 20072 tablet_bootstrap.cc:492] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc: Bootstrap complete.
I20240503 07:59:43.682075 20072 ts_tablet_manager.cc:1367] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:59:43.683539 20072 raft_consensus.cc:348] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:43.683703 20072 raft_consensus.cc:729] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:43.683859 20072 consensus_queue.cc:260] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:43.684165 20072 ts_tablet_manager.cc:1398] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:43.686637 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session 5227b26243484bbfb177a995ceb50afc-b3503f0e1e29430a933986978978f1d4 received from {username='slave'} at 127.0.99.132:55149
I20240503 07:59:43.686735 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session 5227b26243484bbfb177a995ceb50afc-b3503f0e1e29430a933986978978f1d4 on tablet b3503f0e1e29430a933986978978f1d4 with peer 5227b26243484bbfb177a995ceb50afc
I20240503 07:59:43.725705 6888 raft_consensus.cc:1203] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:43.736006 7151 raft_consensus.cc:1203] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 07:59:43.779493 9504 raft_consensus.cc:1203] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:43.840173 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:43.845247 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:43.846174 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:43.855140 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:43.864759 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:43.876048 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:43.879526 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:43.884836 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:43.949843 7151 raft_consensus.cc:1203] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:43.950151 7150 raft_consensus.cc:1203] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:43.981446 7020 raft_consensus.cc:1203] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:43.984671 9504 raft_consensus.cc:1203] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:44.015115 9504 raft_consensus.cc:1203] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 07:59:44.060572 7279 raft_consensus.cc:1203] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:44.089533 6888 raft_consensus.cc:1203] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:44.098381 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:44.103049 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:44.111127 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:44.122995 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:44.125741 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:44.146397 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:44.151075 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:44.153409 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:44.226543 9634 consensus_queue.cc:237] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:44.228775 6888 raft_consensus.cc:1261] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:44.229171 9504 raft_consensus.cc:1261] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:44.229573 19697 consensus_queue.cc:1035] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:44.229718 19765 consensus_queue.cc:1035] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:44.231920 19697 raft_consensus.cc:2932] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:44.232432 6888 raft_consensus.cc:2932] T 96edc0386e0e45a4b9df0f47820c723c P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:44.235641 6740 catalog_manager.cc:5484] T 96edc0386e0e45a4b9df0f47820c723c P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:44.236166 9504 raft_consensus.cc:2932] T 96edc0386e0e45a4b9df0f47820c723c P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:44.244503 9351 tablet_service.cc:1514] Processing DeleteTablet for tablet 96edc0386e0e45a4b9df0f47820c723c with delete_type TABLET_DATA_TOMBSTONED (TS 3befbb2d206f4be89d3e07f38837374b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:44.245424 20206 tablet_replica.cc:330] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:44.245695 20206 raft_consensus.cc:2229] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:44.245901 20206 raft_consensus.cc:2258] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:44.246480 20206 ts_tablet_manager.cc:1875] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:44.249222 20206 ts_tablet_manager.cc:1888] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:44.249375 20206 log.cc:1196] T 96edc0386e0e45a4b9df0f47820c723c P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/96edc0386e0e45a4b9df0f47820c723c
I20240503 07:59:44.256269 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet 96edc0386e0e45a4b9df0f47820c723c (table rebalancer_extra_table_32 [id=62360cc567504bf19c32fc20e93a496f]) successfully deleted
I20240503 07:59:44.397719 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:44.401134 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:44.405351 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:44.405395 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:44.420420 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:44.421272 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:44.426048 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:44.439890 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:44.525715 9767 consensus_queue.cc:237] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:44.527065 7150 raft_consensus.cc:1261] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:44.527110 9504 raft_consensus.cc:1261] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:44.527467 19823 consensus_queue.cc:1035] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:44.527684 19822 consensus_queue.cc:1035] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:44.529667 19822 raft_consensus.cc:2932] T cf3d2c6d70bb49fcb1dd23fc0d402796 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:44.530196 9504 raft_consensus.cc:2932] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:44.530643 7150 raft_consensus.cc:2932] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:44.534420 6738 catalog_manager.cc:5484] T cf3d2c6d70bb49fcb1dd23fc0d402796 P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:44.537415 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3d2c6d70bb49fcb1dd23fc0d402796 with delete_type TABLET_DATA_TOMBSTONED (TS 159177b3caeb434baa98e04136d050b2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:44.538966 20277 tablet_replica.cc:330] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:44.539189 20277 raft_consensus.cc:2229] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:44.539335 20277 raft_consensus.cc:2258] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:44.539834 20277 ts_tablet_manager.cc:1875] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:44.541932 20277 ts_tablet_manager.cc:1888] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:44.542012 20277 log.cc:1196] T cf3d2c6d70bb49fcb1dd23fc0d402796 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/cf3d2c6d70bb49fcb1dd23fc0d402796
I20240503 07:59:44.542557 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet cf3d2c6d70bb49fcb1dd23fc0d402796 (table rebalancer_extra_table_34 [id=eba63fcb11c54ebf8beaa0fe0e7a772a]) successfully deleted
I20240503 07:59:44.677776 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:44.688206 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:44.691358 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:44.694216 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:44.696184 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:44.696512 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:44.698495 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:44.702991 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:44.783231 9374 consensus_queue.cc:237] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:44.784507 9634 raft_consensus.cc:1261] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:44.784902 9505 raft_consensus.cc:1261] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:44.784988 19234 consensus_queue.cc:1035] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:44.785315 18264 consensus_queue.cc:1035] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:44.787228 19234 raft_consensus.cc:2932] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:44.787806 9505 raft_consensus.cc:2932] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:44.791217 9635 raft_consensus.cc:2932] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:44.791306 6746 catalog_manager.cc:5484] T 659657d85679433abdcbe711e4741b67 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:44.795501 9747 tablet_service.cc:1514] Processing DeleteTablet for tablet 659657d85679433abdcbe711e4741b67 with delete_type TABLET_DATA_TOMBSTONED (TS a372e98a670f49e9adca46ab5c836c4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:44.795938 20355 tablet_replica.cc:330] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:44.796161 20355 raft_consensus.cc:2229] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:44.796375 20355 raft_consensus.cc:2258] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:44.797410 20355 ts_tablet_manager.cc:1875] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:44.800009 20355 ts_tablet_manager.cc:1888] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:44.800096 20355 log.cc:1196] T 659657d85679433abdcbe711e4741b67 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/659657d85679433abdcbe711e4741b67
I20240503 07:59:44.800659 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet 659657d85679433abdcbe711e4741b67 (table rebalancer_extra_table_30 [id=ae9c8bb841ff4e139b8fc59887fda0e9]) successfully deleted
I20240503 07:59:44.918067 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:44.921308 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:44.936241 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:44.938134 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:44.938452 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:44.944535 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:44.945250 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:44.965344 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:45.059263 7020 consensus_queue.cc:237] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:45.060825 7151 raft_consensus.cc:1261] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:45.061260 9634 raft_consensus.cc:1261] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:45.061487 19698 consensus_queue.cc:1035] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:45.062152 20330 consensus_queue.cc:1035] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:45.064275 19782 raft_consensus.cc:2932] T 31c76213f3ec451084ae7b0551cd7b48 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:45.065438 9634 raft_consensus.cc:2932] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:45.068323 7150 raft_consensus.cc:2932] T 31c76213f3ec451084ae7b0551cd7b48 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:45.069176 6743 catalog_manager.cc:5484] T 31c76213f3ec451084ae7b0551cd7b48 P 159177b3caeb434baa98e04136d050b2 reported cstate change: config changed from index 6 to 7, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:45.072535 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c76213f3ec451084ae7b0551cd7b48 with delete_type TABLET_DATA_TOMBSTONED (TS 5227b26243484bbfb177a995ceb50afc not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60188
I20240503 07:59:45.072989 20425 tablet_replica.cc:330] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc: stopping tablet replica
I20240503 07:59:45.073216 20425 raft_consensus.cc:2229] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:45.073469 20425 raft_consensus.cc:2258] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:45.074124 20425 ts_tablet_manager.cc:1875] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:45.077404 20425 ts_tablet_manager.cc:1888] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 07:59:45.077559 20425 log.cc:1196] T 31c76213f3ec451084ae7b0551cd7b48 P 5227b26243484bbfb177a995ceb50afc: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/31c76213f3ec451084ae7b0551cd7b48
I20240503 07:59:45.078768 6734 catalog_manager.cc:4830] TS 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): tablet 31c76213f3ec451084ae7b0551cd7b48 (table rebalance_test_table_5 [id=fe06389707474e128f9ddd0adcf4d1c2]) successfully deleted
I20240503 07:59:45.227654 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:45.228891 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:45.233129 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:45.237252 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:45.242493 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:45.243168 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:45.254305 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:45.273200 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:45.340939 9634 consensus_queue.cc:237] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:45.342461 6886 raft_consensus.cc:1261] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:45.342578 9767 raft_consensus.cc:1261] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:45.343281 20490 consensus_queue.cc:1035] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:45.343721 19697 consensus_queue.cc:1035] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:45.345630 20490 raft_consensus.cc:2932] T d54f576fbe3d44ff8311505aef9855e2 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:45.346130 9766 raft_consensus.cc:2932] T d54f576fbe3d44ff8311505aef9855e2 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:45.346529 6886 raft_consensus.cc:2932] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:45.350441 6746 catalog_manager.cc:5484] T d54f576fbe3d44ff8311505aef9855e2 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:45.354382 9351 tablet_service.cc:1514] Processing DeleteTablet for tablet d54f576fbe3d44ff8311505aef9855e2 with delete_type TABLET_DATA_TOMBSTONED (TS 3befbb2d206f4be89d3e07f38837374b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:45.354779 20498 tablet_replica.cc:330] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:45.355753 20498 raft_consensus.cc:2229] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:45.355965 20498 raft_consensus.cc:2258] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:45.356595 20498 ts_tablet_manager.cc:1875] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:45.359818 20498 ts_tablet_manager.cc:1888] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:45.359956 20498 log.cc:1196] T d54f576fbe3d44ff8311505aef9855e2 P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/d54f576fbe3d44ff8311505aef9855e2
I20240503 07:59:45.361872 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet d54f576fbe3d44ff8311505aef9855e2 (table rebalancer_extra_table_30 [id=ae9c8bb841ff4e139b8fc59887fda0e9]) successfully deleted
I20240503 07:59:45.523010 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:45.528999 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:45.532821 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:45.542534 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:45.547760 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:45.558967 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:45.563576 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:45.574524 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:45.671159 7151 consensus_queue.cc:237] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:45.672629 9505 raft_consensus.cc:1261] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:45.672681 9766 raft_consensus.cc:1261] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:45.673045 20563 consensus_queue.cc:1035] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:45.673266 19213 consensus_queue.cc:1035] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:45.675302 20566 raft_consensus.cc:2932] T 16c06ab5a3e54ed2910afa50069810cc P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:45.675724 9505 raft_consensus.cc:2932] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:45.675732 9766 raft_consensus.cc:2932] T 16c06ab5a3e54ed2910afa50069810cc P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:45.680351 6745 catalog_manager.cc:5484] T 16c06ab5a3e54ed2910afa50069810cc P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 6 to 7, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:45.683974 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c06ab5a3e54ed2910afa50069810cc with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:45.685323 20567 tablet_replica.cc:330] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:45.686584 20567 raft_consensus.cc:2229] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:45.686905 20567 raft_consensus.cc:2258] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:45.687531 20567 ts_tablet_manager.cc:1875] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:45.690696 20567 ts_tablet_manager.cc:1888] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 07:59:45.690852 20567 log.cc:1196] T 16c06ab5a3e54ed2910afa50069810cc P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/16c06ab5a3e54ed2910afa50069810cc
I20240503 07:59:45.691614 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 16c06ab5a3e54ed2910afa50069810cc (table rebalance_test_table_15 [id=2054a32b20484bd0998c0e3fc6269184]) successfully deleted
I20240503 07:59:45.830519 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:45.838696 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:45.845590 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:45.845870 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:45.864971 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:45.867383 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:45.869570 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:45.875598 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:45.962275 9373 consensus_queue.cc:237] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:45.964113 7151 raft_consensus.cc:1261] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:45.964334 6886 raft_consensus.cc:1261] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:45.964617 20349 consensus_queue.cc:1035] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:45.964875 19234 consensus_queue.cc:1035] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:45.967048 19234 raft_consensus.cc:2932] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:45.968086 6886 raft_consensus.cc:2932] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:45.970850 6745 catalog_manager.cc:5484] T 96289b8f77bb4241bade14b1011e1de4 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:45.971573 7151 raft_consensus.cc:2932] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:45.975039 9484 tablet_service.cc:1514] Processing DeleteTablet for tablet 96289b8f77bb4241bade14b1011e1de4 with delete_type TABLET_DATA_TOMBSTONED (TS cb73994515af4880b05383e78368bb4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:45.975431 20644 tablet_replica.cc:330] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:45.975670 20644 raft_consensus.cc:2229] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:45.975852 20644 raft_consensus.cc:2258] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:45.976421 20644 ts_tablet_manager.cc:1875] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:45.981114 20644 ts_tablet_manager.cc:1888] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:45.981226 20644 log.cc:1196] T 96289b8f77bb4241bade14b1011e1de4 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/96289b8f77bb4241bade14b1011e1de4
I20240503 07:59:45.981763 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 96289b8f77bb4241bade14b1011e1de4 (table rebalancer_extra_table_2 [id=bcee57aafa9c44d5a92e47a3e072d94b]) successfully deleted
I20240503 07:59:46.127316 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:46.132957 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:46.136849 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:46.138252 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:46.141961 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:46.143813 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:46.163300 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:46.168103 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:46.247558 7150 consensus_queue.cc:237] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:46.248996 7020 raft_consensus.cc:1261] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:46.249058 9634 raft_consensus.cc:1261] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:46.249401 20563 consensus_queue.cc:1035] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:46.249611 20566 consensus_queue.cc:1035] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:46.252673 20563 raft_consensus.cc:2932] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:46.253084 9634 raft_consensus.cc:2932] T b607148b2a7b4b56b83e9606be49221b P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:46.253743 7020 raft_consensus.cc:2932] T b607148b2a7b4b56b83e9606be49221b P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:46.256484 6738 catalog_manager.cc:5484] T b607148b2a7b4b56b83e9606be49221b P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index 2 to 3, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:46.263567 9351 tablet_service.cc:1514] Processing DeleteTablet for tablet b607148b2a7b4b56b83e9606be49221b with delete_type TABLET_DATA_TOMBSTONED (TS 3befbb2d206f4be89d3e07f38837374b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41586
I20240503 07:59:46.264456 20715 tablet_replica.cc:330] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b: stopping tablet replica
I20240503 07:59:46.264701 20715 raft_consensus.cc:2229] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:46.264935 20715 raft_consensus.cc:2258] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:46.265497 20715 ts_tablet_manager.cc:1875] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:46.267726 20715 ts_tablet_manager.cc:1888] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:46.267800 20715 log.cc:1196] T b607148b2a7b4b56b83e9606be49221b P 3befbb2d206f4be89d3e07f38837374b: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/b607148b2a7b4b56b83e9606be49221b
I20240503 07:59:46.268287 6732 catalog_manager.cc:4830] TS 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): tablet b607148b2a7b4b56b83e9606be49221b (table rebalancer_extra_table_22 [id=52f5ffbd2adb439692c9dea927b73abb]) successfully deleted
I20240503 07:59:46.399935 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:46.400219 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:46.406044 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:46.408123 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:46.415092 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:46.439838 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:46.477981 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:46.481478 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:46.529824 9374 consensus_queue.cc:237] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:46.531141 6888 raft_consensus.cc:1261] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:46.531519 7150 raft_consensus.cc:1261] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 3befbb2d206f4be89d3e07f38837374b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:46.531534 18264 consensus_queue.cc:1035] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:46.531879 20349 consensus_queue.cc:1035] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:46.533447 18264 raft_consensus.cc:2932] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:46.533946 6888 raft_consensus.cc:2932] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:46.537523 6747 catalog_manager.cc:5484] T 61b93171848746be89447d2ef90459a7 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "3befbb2d206f4be89d3e07f38837374b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:46.540313 7150 raft_consensus.cc:2932] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:46.542820 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b93171848746be89447d2ef90459a7 with delete_type TABLET_DATA_TOMBSTONED (TS 159177b3caeb434baa98e04136d050b2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:46.543532 20798 tablet_replica.cc:330] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:46.543730 20798 raft_consensus.cc:2229] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:46.543896 20798 raft_consensus.cc:2258] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:46.544413 20798 ts_tablet_manager.cc:1875] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:46.547107 20798 ts_tablet_manager.cc:1888] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:46.547238 20798 log.cc:1196] T 61b93171848746be89447d2ef90459a7 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/61b93171848746be89447d2ef90459a7
I20240503 07:59:46.547804 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 61b93171848746be89447d2ef90459a7 (table rebalancer_extra_table_28 [id=667865e90f934944b36fd3b1fd5f68dc]) successfully deleted
I20240503 07:59:46.573099 9634 consensus_queue.cc:237] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:46.574751 9504 raft_consensus.cc:1261] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.575095 9374 raft_consensus.cc:1261] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.575438 20490 consensus_queue.cc:1035] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:46.575695 20820 consensus_queue.cc:1035] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:46.575922 9572 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): Couldn't send request to peer 5227b26243484bbfb177a995ceb50afc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: faee1f8d939e4487a014044772ba2daf. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:46.577668 20820 raft_consensus.cc:2932] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:46.578090 9504 raft_consensus.cc:2932] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:46.582420 9374 raft_consensus.cc:2932] T faee1f8d939e4487a014044772ba2daf P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:46.582561 6741 catalog_manager.cc:5484] T faee1f8d939e4487a014044772ba2daf P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:46.605540 9634 consensus_queue.cc:237] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:46.607062 9374 raft_consensus.cc:1261] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:46.607357 9571 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): Couldn't send request to peer 9c085a3241f14cb7ba1332b508f4b57d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fef7df80245c4b2c9fd3efd8331ba9e4. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:46.607497 19765 consensus_queue.cc:1035] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:46.607590 7279 raft_consensus.cc:1261] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.607968 19765 consensus_queue.cc:1035] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:46.611047 19765 raft_consensus.cc:2932] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:46.611470 9374 raft_consensus.cc:2932] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:46.613468 7279 raft_consensus.cc:2932] T fef7df80245c4b2c9fd3efd8331ba9e4 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:46.615245 6747 catalog_manager.cc:5484] T fef7df80245c4b2c9fd3efd8331ba9e4 P 3befbb2d206f4be89d3e07f38837374b reported cstate change: config changed from index -1 to 2, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:46.769650 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:46.775058 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:46.783875 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:46.790594 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:46.792054 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:46.801138 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:46.835871 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:46.839100 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:47.029883 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:47.031862 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:47.032361 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:47.040477 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:47.046940 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:47.051990 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:47.072821 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:47.098598 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:47.128224 20948 ts_tablet_manager.cc:902] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc: Initiating tablet copy from peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:47.128948 20948 tablet_copy_client.cc:308] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:43369
I20240503 07:59:47.129560 9656 tablet_copy_service.cc:140] P 159177b3caeb434baa98e04136d050b2: Received BeginTabletCopySession request for tablet faee1f8d939e4487a014044772ba2daf from peer 5227b26243484bbfb177a995ceb50afc ({username='slave'} at 127.0.99.132:56797)
I20240503 07:59:47.129695 9656 tablet_copy_service.cc:161] P 159177b3caeb434baa98e04136d050b2: Beginning new tablet copy session on tablet faee1f8d939e4487a014044772ba2daf from peer 5227b26243484bbfb177a995ceb50afc at {username='slave'} at 127.0.99.132:56797: session id = 5227b26243484bbfb177a995ceb50afc-faee1f8d939e4487a014044772ba2daf
I20240503 07:59:47.130846 9656 tablet_copy_source_session.cc:205] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:47.132755 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faee1f8d939e4487a014044772ba2daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.135814 20948 tablet_copy_client.cc:783] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:47.135995 20948 tablet_copy_client.cc:647] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:47.137759 20948 tablet_copy_client.cc:523] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:47.139624 20948 tablet_bootstrap.cc:492] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
I20240503 07:59:47.144232 20948 tablet_bootstrap.cc:492] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:47.144484 20948 tablet_bootstrap.cc:492] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc: Bootstrap complete.
I20240503 07:59:47.144665 20948 ts_tablet_manager.cc:1367] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 07:59:47.144914 20965 ts_tablet_manager.cc:902] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d: Initiating tablet copy from peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:47.145207 20948 raft_consensus.cc:348] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:47.145349 20948 raft_consensus.cc:729] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.145445 20965 tablet_copy_client.cc:308] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:43369
I20240503 07:59:47.145496 20948 consensus_queue.cc:260] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:47.145813 20948 ts_tablet_manager.cc:1398] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:47.145932 9656 tablet_copy_service.cc:140] P 159177b3caeb434baa98e04136d050b2: Received BeginTabletCopySession request for tablet fef7df80245c4b2c9fd3efd8331ba9e4 from peer 9c085a3241f14cb7ba1332b508f4b57d ({username='slave'} at 127.0.99.130:60049)
I20240503 07:59:47.146040 9656 tablet_copy_service.cc:161] P 159177b3caeb434baa98e04136d050b2: Beginning new tablet copy session on tablet fef7df80245c4b2c9fd3efd8331ba9e4 from peer 9c085a3241f14cb7ba1332b508f4b57d at {username='slave'} at 127.0.99.130:60049: session id = 9c085a3241f14cb7ba1332b508f4b57d-fef7df80245c4b2c9fd3efd8331ba9e4
I20240503 07:59:47.146371 9655 tablet_copy_service.cc:342] P 159177b3caeb434baa98e04136d050b2: Request end of tablet copy session 5227b26243484bbfb177a995ceb50afc-faee1f8d939e4487a014044772ba2daf received from {username='slave'} at 127.0.99.132:56797
I20240503 07:59:47.146452 9655 tablet_copy_service.cc:433] P 159177b3caeb434baa98e04136d050b2: ending tablet copy session 5227b26243484bbfb177a995ceb50afc-faee1f8d939e4487a014044772ba2daf on tablet faee1f8d939e4487a014044772ba2daf with peer 5227b26243484bbfb177a995ceb50afc
I20240503 07:59:47.147189 9656 tablet_copy_source_session.cc:205] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:47.148133 20965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef7df80245c4b2c9fd3efd8331ba9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.150707 20965 tablet_copy_client.cc:783] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:47.150875 20965 tablet_copy_client.cc:647] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:47.152031 20965 tablet_copy_client.cc:523] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:47.153487 20965 tablet_bootstrap.cc:492] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap starting.
I20240503 07:59:47.158387 20965 tablet_bootstrap.cc:492] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:47.158674 20965 tablet_bootstrap.cc:492] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d: Bootstrap complete.
I20240503 07:59:47.158869 20965 ts_tablet_manager.cc:1367] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20240503 07:59:47.159329 20965 raft_consensus.cc:348] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:47.159440 20965 raft_consensus.cc:729] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.159554 20965 consensus_queue.cc:260] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:47.159842 20965 ts_tablet_manager.cc:1398] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:47.160789 9656 tablet_copy_service.cc:342] P 159177b3caeb434baa98e04136d050b2: Request end of tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-fef7df80245c4b2c9fd3efd8331ba9e4 received from {username='slave'} at 127.0.99.130:60049
I20240503 07:59:47.160881 9656 tablet_copy_service.cc:433] P 159177b3caeb434baa98e04136d050b2: ending tablet copy session 9c085a3241f14cb7ba1332b508f4b57d-fef7df80245c4b2c9fd3efd8331ba9e4 on tablet fef7df80245c4b2c9fd3efd8331ba9e4 with peer 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:47.295173 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:47.306180 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:47.308611 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:47.308766 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:47.309763 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:47.310021 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:47.349236 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:47.354967 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:47.427706 9505 consensus_queue.cc:237] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:47.429206 9373 raft_consensus.cc:1261] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:47.429289 7151 raft_consensus.cc:1261] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer cb73994515af4880b05383e78368bb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:47.429672 19381 consensus_queue.cc:1035] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:47.432456 17600 raft_consensus.cc:2932] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:47.433854 19381 consensus_queue.cc:1035] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:47.434092 9373 raft_consensus.cc:2932] T 9cfe98aefcfe4380bb64f14df1a24318 P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:47.435350 7150 raft_consensus.cc:2932] T 9cfe98aefcfe4380bb64f14df1a24318 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:47.436785 6738 catalog_manager.cc:5484] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 2 to 3, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "cb73994515af4880b05383e78368bb4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:47.443280 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cfe98aefcfe4380bb64f14df1a24318 with delete_type TABLET_DATA_TOMBSTONED (TS 159177b3caeb434baa98e04136d050b2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46988
I20240503 07:59:47.443966 21038 tablet_replica.cc:330] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2: stopping tablet replica
I20240503 07:59:47.444258 21038 raft_consensus.cc:2229] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:47.444468 21038 raft_consensus.cc:2258] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:47.445003 21038 ts_tablet_manager.cc:1875] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:47.448959 21038 ts_tablet_manager.cc:1888] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:47.449167 21038 log.cc:1196] T 9cfe98aefcfe4380bb64f14df1a24318 P 159177b3caeb434baa98e04136d050b2: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/9cfe98aefcfe4380bb64f14df1a24318
I20240503 07:59:47.449831 6731 catalog_manager.cc:4830] TS 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): tablet 9cfe98aefcfe4380bb64f14df1a24318 (table rebalancer_extra_table_16 [id=a0ee201ab5cc47b798438b4276757b44]) successfully deleted
I20240503 07:59:47.520027 7020 raft_consensus.cc:1203] T fef7df80245c4b2c9fd3efd8331ba9e4 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:47.584705 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:47.590185 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:47.599891 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:47.600515 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:47.604974 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:47.613600 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:47.619841 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:47.643280 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:47.706571 7151 consensus_queue.cc:237] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:47.707954 9505 raft_consensus.cc:1261] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:47.708050 7282 raft_consensus.cc:1261] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:47.708406 21105 consensus_queue.cc:1035] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:47.708628 20905 consensus_queue.cc:1035] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:47.710503 21108 raft_consensus.cc:2932] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:47.710901 7282 raft_consensus.cc:2932] T 660355a4482d4a30bbb71cd0b56527d1 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:47.710868 9505 raft_consensus.cc:2932] T 660355a4482d4a30bbb71cd0b56527d1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:47.714726 6747 catalog_manager.cc:5484] T 660355a4482d4a30bbb71cd0b56527d1 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index 4 to 5, VOTER 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:47.717922 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 660355a4482d4a30bbb71cd0b56527d1 with delete_type TABLET_DATA_TOMBSTONED (TS 9c085a3241f14cb7ba1332b508f4b57d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35192
I20240503 07:59:47.718309 21109 tablet_replica.cc:330] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d: stopping tablet replica
I20240503 07:59:47.718502 21109 raft_consensus.cc:2229] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:47.718866 21109 raft_consensus.cc:2258] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:47.719398 21109 ts_tablet_manager.cc:1875] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:47.721853 21109 ts_tablet_manager.cc:1888] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:47.721980 21109 log.cc:1196] T 660355a4482d4a30bbb71cd0b56527d1 P 9c085a3241f14cb7ba1332b508f4b57d: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/660355a4482d4a30bbb71cd0b56527d1
I20240503 07:59:47.722993 6733 catalog_manager.cc:4830] TS 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383): tablet 660355a4482d4a30bbb71cd0b56527d1 (table rebalance_test_table_1 [id=7240d8c6aeff4067b4b95979e77f21da]) successfully deleted
I20240503 07:59:47.725682 7282 raft_consensus.cc:1203] T faee1f8d939e4487a014044772ba2daf P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:47.853533 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:47.854233 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:47.866815 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:47.868793 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:47.869817 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:47.876138 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:47.877588 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:47.877585 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:47.974151 6888 consensus_queue.cc:237] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:47.975690 7282 raft_consensus.cc:1261] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:47.975703 9635 raft_consensus.cc:1261] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:47.976147 19454 consensus_queue.cc:1035] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:47.976557 19454 consensus_queue.cc:1035] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:47.978869 19454 raft_consensus.cc:2932] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:47.981091 7282 raft_consensus.cc:2932] T b3503f0e1e29430a933986978978f1d4 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:47.982517 9635 raft_consensus.cc:2932] T b3503f0e1e29430a933986978978f1d4 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:47.988368 6747 catalog_manager.cc:5484] T b3503f0e1e29430a933986978978f1d4 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index 2 to 3, VOTER a372e98a670f49e9adca46ab5c836c4a (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:47.993196 9747 tablet_service.cc:1514] Processing DeleteTablet for tablet b3503f0e1e29430a933986978978f1d4 with delete_type TABLET_DATA_TOMBSTONED (TS a372e98a670f49e9adca46ab5c836c4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59044
I20240503 07:59:47.996892 21186 tablet_replica.cc:330] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a: stopping tablet replica
I20240503 07:59:47.997087 21186 raft_consensus.cc:2229] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:47.997232 21186 raft_consensus.cc:2258] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:47.997896 21186 ts_tablet_manager.cc:1875] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:48.000775 21186 ts_tablet_manager.cc:1888] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:48.000866 21186 log.cc:1196] T b3503f0e1e29430a933986978978f1d4 P a372e98a670f49e9adca46ab5c836c4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/b3503f0e1e29430a933986978978f1d4
I20240503 07:59:48.001639 6734 catalog_manager.cc:4830] TS a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683): tablet b3503f0e1e29430a933986978978f1d4 (table rebalancer_extra_table_14 [id=f0fcbc958ed14af18b82b7bb244d2bfb]) successfully deleted
I20240503 07:59:48.082484 9767 consensus_queue.cc:237] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
W20240503 07:59:48.084096 9703 consensus_peers.cc:485] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a -> Peer 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): Couldn't send request to peer 00a3f8e159c64327832d79fb32cdcca8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23de5c1fe8c44b3c990f1128826a72b0. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:48.084120 9504 raft_consensus.cc:1261] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.084250 7020 raft_consensus.cc:1261] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.084576 19823 consensus_queue.cc:1035] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:48.084815 21088 consensus_queue.cc:1035] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:48.087167 21088 raft_consensus.cc:2932] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:48.087615 7020 raft_consensus.cc:2932] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:48.089258 9504 raft_consensus.cc:2932] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:48.091184 6739 catalog_manager.cc:5484] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index -1 to 2, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:48.117650 7282 consensus_queue.cc:237] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:48.119479 7150 raft_consensus.cc:1261] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:48.119859 7020 raft_consensus.cc:1261] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:48.119864 21262 consensus_queue.cc:1035] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:48.120227 21262 consensus_queue.cc:1035] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:48.120468 7215 consensus_peers.cc:485] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ae7895f7d7f47c3ab5fb0b93c1fd8e0. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:48.122596 20078 raft_consensus.cc:2932] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:48.123090 7150 raft_consensus.cc:2932] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:48.126060 7020 raft_consensus.cc:2932] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:48.127121 6744 catalog_manager.cc:5484] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index -1 to 4, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:48.150876 9767 consensus_queue.cc:237] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:48.152616 9504 raft_consensus.cc:1261] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.152750 9373 raft_consensus.cc:1261] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:48.153090 9703 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483): Couldn't send request to peer 5227b26243484bbfb177a995ceb50afc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbc7505d3ef6416c9d505bfe899f73ed. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:48.153205 21245 consensus_queue.cc:1035] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:48.153399 21088 consensus_queue.cc:1035] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:48.155531 21245 raft_consensus.cc:2932] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:48.155977 9373 raft_consensus.cc:2932] T fbc7505d3ef6416c9d505bfe899f73ed P 3befbb2d206f4be89d3e07f38837374b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:48.157824 9504 raft_consensus.cc:2932] T fbc7505d3ef6416c9d505bfe899f73ed P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:48.159932 6745 catalog_manager.cc:5484] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index -1 to 2, VOTER 5227b26243484bbfb177a995ceb50afc (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:48.181960 6886 consensus_queue.cc:237] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:48.184338 9767 raft_consensus.cc:1261] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:48.184444 7150 raft_consensus.cc:1261] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 07:59:48.184723 6822 consensus_peers.cc:485] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369): Couldn't send request to peer 159177b3caeb434baa98e04136d050b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b794508aa77492398e48d25edfb8aa3. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:48.184769 19454 consensus_queue.cc:1035] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:48.185163 19454 consensus_queue.cc:1035] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:48.187094 21180 raft_consensus.cc:2932] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:48.187512 7150 raft_consensus.cc:2932] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:48.190335 9767 raft_consensus.cc:2932] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:48.191141 6739 catalog_manager.cc:5484] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index 5 to 6, VOTER 159177b3caeb434baa98e04136d050b2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:48.219769 7282 consensus_queue.cc:237] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:48.221576 7150 raft_consensus.cc:1261] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 07:59:48.221783 7216 consensus_peers.cc:485] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd6aa70d9c4c4baea597bbb9788fdb37. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:48.221979 21262 consensus_queue.cc:1035] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:48.230576 9504 raft_consensus.cc:1261] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:48.231055 21262 consensus_queue.cc:1035] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:48.233079 21262 raft_consensus.cc:2932] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:48.233563 9504 raft_consensus.cc:2932] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:48.237426 6747 catalog_manager.cc:5484] T dd6aa70d9c4c4baea597bbb9788fdb37 P cb73994515af4880b05383e78368bb4a reported cstate change: config changed from index 5 to 6, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:48.238189 7150 raft_consensus.cc:2932] T dd6aa70d9c4c4baea597bbb9788fdb37 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:48.372860 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:48.381842 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:48.393949 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:48.394747 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:48.401166 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:48.402516 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:48.413319 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:48.443614 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:48.547716 21415 ts_tablet_manager.cc:902] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:48.548377 21415 tablet_copy_client.cc:308] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37483
I20240503 07:59:48.549162 7302 tablet_copy_service.cc:140] P 5227b26243484bbfb177a995ceb50afc: Received BeginTabletCopySession request for tablet 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:43259)
I20240503 07:59:48.549279 7302 tablet_copy_service.cc:161] P 5227b26243484bbfb177a995ceb50afc: Beginning new tablet copy session on tablet 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:43259: session id = 159177b3caeb434baa98e04136d050b2-1ae7895f7d7f47c3ab5fb0b93c1fd8e0
I20240503 07:59:48.550221 7302 tablet_copy_source_session.cc:205] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:48.551286 21415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae7895f7d7f47c3ab5fb0b93c1fd8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.553910 21415 tablet_copy_client.cc:783] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:48.554100 21415 tablet_copy_client.cc:647] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:48.560212 21415 tablet_copy_client.cc:523] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:48.562376 21415 tablet_bootstrap.cc:492] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:48.590668 21415 tablet_bootstrap.cc:492] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:48.591006 21415 tablet_bootstrap.cc:492] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:48.591285 21415 ts_tablet_manager.cc:1367] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.000s
I20240503 07:59:48.591833 21415 raft_consensus.cc:348] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:48.592000 21415 raft_consensus.cc:729] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.592455 21415 consensus_queue.cc:260] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:48.593083 21415 ts_tablet_manager.cc:1398] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:48.594038 7302 tablet_copy_service.cc:342] P 5227b26243484bbfb177a995ceb50afc: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-1ae7895f7d7f47c3ab5fb0b93c1fd8e0 received from {username='slave'} at 127.0.99.135:43259
I20240503 07:59:48.594132 7302 tablet_copy_service.cc:433] P 5227b26243484bbfb177a995ceb50afc: ending tablet copy session 159177b3caeb434baa98e04136d050b2-1ae7895f7d7f47c3ab5fb0b93c1fd8e0 on tablet 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 with peer 159177b3caeb434baa98e04136d050b2
I20240503 07:59:48.637553 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:48.642450 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:48.651329 9351 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:48.656666 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:48.660034 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:48.667964 21432 ts_tablet_manager.cc:902] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8: Initiating tablet copy from peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:48.668602 21432 tablet_copy_client.cc:308] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:43683
I20240503 07:59:48.673035 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:48.680444 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:48.680680 9787 tablet_copy_service.cc:140] P a372e98a670f49e9adca46ab5c836c4a: Received BeginTabletCopySession request for tablet 23de5c1fe8c44b3c990f1128826a72b0 from peer 00a3f8e159c64327832d79fb32cdcca8 ({username='slave'} at 127.0.99.131:49907)
I20240503 07:59:48.682264 9787 tablet_copy_service.cc:161] P a372e98a670f49e9adca46ab5c836c4a: Beginning new tablet copy session on tablet 23de5c1fe8c44b3c990f1128826a72b0 from peer 00a3f8e159c64327832d79fb32cdcca8 at {username='slave'} at 127.0.99.131:49907: session id = 00a3f8e159c64327832d79fb32cdcca8-23de5c1fe8c44b3c990f1128826a72b0
I20240503 07:59:48.683337 9787 tablet_copy_source_session.cc:205] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:48.685902 21433 ts_tablet_manager.cc:902] T dd6aa70d9c4c4baea597bbb9788fdb37 P 3befbb2d206f4be89d3e07f38837374b: Initiating tablet copy from peer 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
F20240503 07:59:48.686159 21433 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-5293a26b65484ae5a9e76469e644c5bc.queue_time_us, tablet-download-pool-5293a26b65484ae5a9e76469e644c5bc.queue_time_us) (tablet-download-pool-5293a26b65484ae5a9e76469e644c5bc.run_cpu_time_us, tablet-download-pool-5293a26b65484ae5a9e76469e644c5bc.run_cpu_time_us) (tablet-download-pool-5293a26b65484ae5a9e76469e644c5bc.run_wall_time_us, tablet-download-pool-5293a26b65484ae5a9e76469e644c5bc.run_wall_time_us) (tablet-download-pool-5365969434f142b8a5757a9a73bc109a.queue_time_us, tablet-download-pool-5365969434f142b8a5757a9a73bc109a.queue_time_us) (tablet-download-pool-5365969434f142b8a5757a9a73bc109a.run_cpu_time_us, tablet-download-pool-5365969434f142b8a5757a9a73bc109a.run_cpu_time_us) (tablet-download-pool-5365969434f142b8a5757a9a73bc109a.run_wall_time_us, tablet-download-pool-5365969434f142b8a5757a9a73bc109a.run_wall_time_us) (tablet-download-pool-5388c1fa25bc4f19a2aef9d8960e88aa.queue_time_us, tablet-download-pool-5388c1fa25bc4f19a2aef9d8960e88aa.queue_time_us) (tablet-download-pool-5388c1fa25bc4f19a2aef9d8960e88aa.run_cpu_time_us, tablet-download-pool-5388c1fa25bc4f19a2aef9d8960e88aa.run_cpu_time_us) (tablet-download-pool-5388c1fa25bc4f19a2aef9d8960e88aa.run_wall_time_us, tablet-download-pool-5388c1fa25bc4f19a2aef9d8960e88aa.run_wall_time_us) (tablet-download-pool-5390e70b0449449aa3af05112bd29797.queue_time_us, tablet-download-pool-5390e70b0449449aa3af05112bd29797.queue_time_us) (tablet-download-pool-5390e70b0449449aa3af05112bd29797.run_cpu_time_us, tablet-download-pool-5390e70b0449449aa3af05112bd29797.run_cpu_time_us) (tablet-download-pool-5390e70b0449449aa3af05112bd29797.run_wall_time_us, tablet-download-pool-5390e70b0449449aa3af05112bd29797.run_wall_time_us) (tablet-download-pool-5cbcf7ab42a1463582e96927d7f16b9a.queue_time_us, tablet-download-pool-5cbcf7ab42a1463582e96927d7f16b9a.queue_time_us) (tablet-download-pool-5cbcf7ab42a1463582e96927d7f16b9a.run_cpu_time_us, tablet-download-pool-5cbcf7ab42a1463582e96927d7f16b9a.run_cpu_time_us) (tablet-download-pool-5cbcf7ab42a1463582e96927d7f16b9a.run_wall_time_us, tablet-download-pool-5cbcf7ab42a1463582e96927d7f16b9a.run_wall_time_us) (tablet-download-pool-6e63db30c0454722b6a794a9259a67c0.queue_time_us, tablet-download-pool-6e63db30c0454722b6a794a9259a67c0.queue_time_us) (tablet-download-pool-6e63db30c0454722b6a794a9259a67c0.run_cpu_time_us, tablet-download-pool-6e63db30c0454722b6a794a9259a67c0.run_cpu_time_us) (tablet-download-pool-6e63db30c0454722b6a794a9259a67c0.run_wall_time_us, tablet-download-pool-6e63db30c0454722b6a794a9259a67c0.run_wall_time_us) (tablet-download-pool-721c34a8b6a348e5890d462ba7da938c.queue_time_us, tablet-download-pool-721c34a8b6a348e5890d462ba7da938c.queue_time_us) (tablet-download-pool-721c34a8b6a348e5890d462ba7da938c.run_cpu_time_us, tablet-download-pool-721c34a8b6a348e5890d462ba7da938c.run_cpu_time_us) (tablet-download-pool-721c34a8b6a348e5890d462ba7da938c.run_wall_time_us, tablet-download-pool-721c34a8b6a348e5890d462ba7da938c.run_wall_time_us) (tablet-download-pool-795f69249afe444ba0828875e1839b8c.queue_time_us, tablet-download-pool-795f69249afe444ba0828875e1839b8c.queue_time_us) (tablet-download-pool-795f69249afe444ba0828875e1839b8c.run_cpu_time_us, tablet-download-pool-795f69249afe444ba0828875e1839b8c.run_cpu_time_us) (tablet-download-pool-795f69249afe444ba0828875e1839b8c.run_wall_time_us, tablet-download-pool-795f69249afe444ba0828875e1839b8c.run_wall_time_us) (tablet-download-pool-7d3eb284c0fc48b4912168c434844284.queue_time_us, tablet-download-pool-7d3eb284c0fc48b4912168c434844284.queue_time_us) (tablet-download-pool-7d3eb284c0fc48b4912168c434844284.run_cpu_time_us, tablet-download-pool-7d3eb284c0fc48b4912168c434844284.run_cpu_time_us) (tablet-download-pool-7d3eb284c0fc48b4912168c434844284.run_wall_time_us, tablet-download-pool-7d3eb284c0fc48b4912168c434844284.run_wall_time_us) (tablet-download-pool-7f946926ddd84a2cb758a795638e3ef5.queue_time_us, tablet-download-pool-7f946926ddd84a2cb758a795638e3ef5.queue_time_us) (tablet-download-pool-7f946926ddd84a2cb758a795638e3ef5.run_cpu_time_us, tablet-download-pool-7f946926ddd84a2cb758a795638e3ef5.run_cpu_time_us) (tablet-download-pool-7f946926ddd84a2cb758a795638e3ef5.run_wall_time_us, tablet-download-pool-7f946926ddd84a2cb758a795638e3ef5.run_wall_time_us) (tablet-download-pool-abb55c09be324f37a1a546e73008a80d.queue_time_us, tablet-download-pool-abb55c09be324f37a1a546e73008a80d.queue_time_us) (tablet-download-pool-abb55c09be324f37a1a546e73008a80d.run_cpu_time_us, tablet-download-pool-abb55c09be324f37a1a546e73008a80d.run_cpu_time_us) (tablet-download-pool-abb55c09be324f37a1a546e73008a80d.run_wall_time_us, tablet-download-pool-abb55c09be324f37a1a546e73008a80d.run_wall_time_us) (tablet-download-pool-b74fd6f87ab44bebb22167d0b6372cd1.queue_time_us, tablet-download-pool-b74fd6f87ab44bebb22167d0b6372cd1.queue_time_us) (tablet-download-pool-b74fd6f87ab44bebb22167d0b6372cd1.run_cpu_time_us, tablet-download-pool-b74fd6f87ab44bebb22167d0b6372cd1.run_cpu_time_us) (tablet-download-pool-b74fd6f87ab44bebb22167d0b6372cd1.run_wall_time_us, tablet-download-pool-b74fd6f87ab44bebb22167d0b6372cd1.run_wall_time_us) (tablet-download-pool-b916e89edaf046b99767e5cca531bb8b.queue_time_us, tablet-download-pool-b916e89edaf046b99767e5cca531bb8b.queue_time_us) (tablet-download-pool-b916e89edaf046b99767e5cca531bb8b.run_cpu_time_us, tablet-download-pool-b916e89edaf046b99767e5cca531bb8b.run_cpu_time_us) (tablet-download-pool-b916e89edaf046b99767e5cca531bb8b.run_wall_time_us, tablet-download-pool-b916e89edaf046b99767e5cca531bb8b.run_wall_time_us) (tablet-download-pool-dd6aa70d9c4c4baea597bbb9788fdb37.queue_time_us, tablet-download-pool-dd6aa70d9c4c4baea597bbb9788fdb37.queue_time_us) (tablet-download-pool-fb1bff0a8228454f95c14762536a4c82.queue_time_us, tablet-download-pool-fb1bff0a8228454f95c14762536a4c82.queue_time_us) (tablet-download-pool-fb1bff0a8228454f95c14762536a4c82.run_cpu_time_us, tablet-download-pool-fb1bff0a8228454f95c14762536a4c82.run_cpu_time_us) (tablet-download-pool-fb1bff0a8228454f95c14762536a4c82.run_wall_time_us, tablet-download-pool-fb1bff0a8228454f95c14762536a4c82.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 07:59:48.691234 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
@ 0x7fdffbfebdcd google::LogMessage::Fail() at ??:0
I20240503 07:59:48.711053 21415 ts_tablet_manager.cc:902] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2: Initiating tablet copy from peer c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:48.711547 21415 tablet_copy_client.cc:308] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40655
I20240503 07:59:48.711884 21432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23de5c1fe8c44b3c990f1128826a72b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.712703 6908 tablet_copy_service.cc:140] P c4218904dd7e42d2be95b901a6e5db92: Received BeginTabletCopySession request for tablet 5b794508aa77492398e48d25edfb8aa3 from peer 159177b3caeb434baa98e04136d050b2 ({username='slave'} at 127.0.99.135:59233)
I20240503 07:59:48.712812 6908 tablet_copy_service.cc:161] P c4218904dd7e42d2be95b901a6e5db92: Beginning new tablet copy session on tablet 5b794508aa77492398e48d25edfb8aa3 from peer 159177b3caeb434baa98e04136d050b2 at {username='slave'} at 127.0.99.135:59233: session id = 159177b3caeb434baa98e04136d050b2-5b794508aa77492398e48d25edfb8aa3
I20240503 07:59:48.713930 6908 tablet_copy_source_session.cc:205] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:48.715039 21415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b794508aa77492398e48d25edfb8aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.717770 21415 tablet_copy_client.cc:783] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:48.717933 21415 tablet_copy_client.cc:647] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:48.720748 21415 tablet_copy_client.cc:523] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7fdffbfefb93 google::LogMessage::SendToLog() at ??:0
I20240503 07:59:48.722793 21415 tablet_bootstrap.cc:492] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2: Bootstrap starting.
I20240503 07:59:48.722883 21432 tablet_copy_client.cc:783] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:48.723047 21432 tablet_copy_client.cc:647] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:48.727988 21432 tablet_copy_client.cc:523] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:48.729727 21432 tablet_bootstrap.cc:492] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap starting.
@ 0x7fdffbfeb7cc google::LogMessage::Flush() at ??:0
@ 0x7fdffbfecf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdffd455044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fdffd439f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fdffd437efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fdffffea77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 07:59:48.764842 21432 tablet_bootstrap.cc:492] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:48.765182 21432 tablet_bootstrap.cc:492] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8: Bootstrap complete.
@ 0x7fdfffff3a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 07:59:48.765383 21432 ts_tablet_manager.cc:1367] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.001s
I20240503 07:59:48.765951 21432 raft_consensus.cc:348] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:48.766994 21432 raft_consensus.cc:729] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.767174 21432 consensus_queue.cc:260] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:48.767606 21432 ts_tablet_manager.cc:1398] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 07:59:48.768335 9787 tablet_copy_service.cc:342] P a372e98a670f49e9adca46ab5c836c4a: Request end of tablet copy session 00a3f8e159c64327832d79fb32cdcca8-23de5c1fe8c44b3c990f1128826a72b0 received from {username='slave'} at 127.0.99.131:49907
I20240503 07:59:48.768420 9787 tablet_copy_service.cc:433] P a372e98a670f49e9adca46ab5c836c4a: ending tablet copy session 00a3f8e159c64327832d79fb32cdcca8-23de5c1fe8c44b3c990f1128826a72b0 on tablet 23de5c1fe8c44b3c990f1128826a72b0 with peer 00a3f8e159c64327832d79fb32cdcca8
@ 0x7fe00007476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 07:59:48.772675 21415 tablet_bootstrap.cc:492] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:48.772977 21415 tablet_bootstrap.cc:492] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2: Bootstrap complete.
I20240503 07:59:48.773155 21415 ts_tablet_manager.cc:1367] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2: Time spent bootstrapping tablet: real 0.050s user 0.025s sys 0.000s
I20240503 07:59:48.773716 21415 raft_consensus.cc:348] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:48.773846 21415 raft_consensus.cc:729] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.774020 21415 consensus_queue.cc:260] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:48.774335 21415 ts_tablet_manager.cc:1398] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7fe0000884f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 07:59:48.775137 6908 tablet_copy_service.cc:342] P c4218904dd7e42d2be95b901a6e5db92: Request end of tablet copy session 159177b3caeb434baa98e04136d050b2-5b794508aa77492398e48d25edfb8aa3 received from {username='slave'} at 127.0.99.135:59233
I20240503 07:59:48.775234 6908 tablet_copy_service.cc:433] P c4218904dd7e42d2be95b901a6e5db92: ending tablet copy session 159177b3caeb434baa98e04136d050b2-5b794508aa77492398e48d25edfb8aa3 on tablet 5b794508aa77492398e48d25edfb8aa3 with peer 159177b3caeb434baa98e04136d050b2
@ 0x7fe000073226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 07:59:48.780896 21446 ts_tablet_manager.cc:902] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc: Initiating tablet copy from peer a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:48.781535 21446 tablet_copy_client.cc:308] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:43683
@ 0x7fe000082c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240503 07:59:48.782125 9787 tablet_copy_service.cc:140] P a372e98a670f49e9adca46ab5c836c4a: Received BeginTabletCopySession request for tablet fbc7505d3ef6416c9d505bfe899f73ed from peer 5227b26243484bbfb177a995ceb50afc ({username='slave'} at 127.0.99.132:37701)
I20240503 07:59:48.782243 9787 tablet_copy_service.cc:161] P a372e98a670f49e9adca46ab5c836c4a: Beginning new tablet copy session on tablet fbc7505d3ef6416c9d505bfe899f73ed from peer 5227b26243484bbfb177a995ceb50afc at {username='slave'} at 127.0.99.132:37701: session id = 5227b26243484bbfb177a995ceb50afc-fbc7505d3ef6416c9d505bfe899f73ed
I20240503 07:59:48.783313 9787 tablet_copy_source_session.cc:205] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x55e8a24e475c std::function<>::operator()() at ??:0
I20240503 07:59:48.786324 21446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc7505d3ef6416c9d505bfe899f73ed. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fdffd43ce06 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 07:59:48.788615 21446 tablet_copy_client.cc:783] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:48.788743 21446 tablet_copy_client.cc:647] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc: tablet copy: Starting download of 1 WAL segments...
@ 0x7fdffd43d6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 07:59:48.789933 21446 tablet_copy_client.cc:523] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7fdffd43f03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240503 07:59:48.791568 21446 tablet_bootstrap.cc:492] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc: Bootstrap starting.
@ 0x55e8a24e475c std::function<>::operator()() at ??:0
I20240503 07:59:48.795774 21446 tablet_bootstrap.cc:492] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
@ 0x7fdffd42c807 kudu::Thread::SuperviseThread() at ??:0
I20240503 07:59:48.797544 21446 tablet_bootstrap.cc:492] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc: Bootstrap complete.
I20240503 07:59:48.797984 21446 ts_tablet_manager.cc:1367] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
@ 0x7fdffdc6a6db start_thread at ??:0
I20240503 07:59:48.799013 21446 raft_consensus.cc:348] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:48.799144 21446 raft_consensus.cc:729] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.799268 21446 consensus_queue.cc:260] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:48.799556 21446 ts_tablet_manager.cc:1398] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
@ 0x7fdffb1a271f clone at ??:0
I20240503 07:59:48.801506 9787 tablet_copy_service.cc:342] P a372e98a670f49e9adca46ab5c836c4a: Request end of tablet copy session 5227b26243484bbfb177a995ceb50afc-fbc7505d3ef6416c9d505bfe899f73ed received from {username='slave'} at 127.0.99.132:37701
I20240503 07:59:48.801586 9787 tablet_copy_service.cc:433] P a372e98a670f49e9adca46ab5c836c4a: ending tablet copy session 5227b26243484bbfb177a995ceb50afc-fbc7505d3ef6416c9d505bfe899f73ed on tablet fbc7505d3ef6416c9d505bfe899f73ed with peer 5227b26243484bbfb177a995ceb50afc
I20240503 07:59:48.980710 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:48.991343 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:48.997071 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:48.997355 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:49.004688 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:49.011492 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:49.013023 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 07:59:49.127694 6954 connection.cc:535] client connection to 127.0.99.133:37815 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:49.127665 9439 connection.cc:535] client connection to 127.0.99.133:37815 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:49.127738 9570 connection.cc:535] client connection to 127.0.99.133:37815 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:49.127847 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:49.127847 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:49.127848 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:49.127864 7085 connection.cc:535] client connection to 127.0.99.133:37815 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:49.127954 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:49.128559 7216 connection.cc:535] client connection to 127.0.99.133:37815 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:49.128687 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:49.128732 9701 connection.cc:535] client connection to 127.0.99.133:37815 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:49.128842 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:49.130105 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:49.130182 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.131107 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.131184 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.131569 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.131608 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.131664 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.131700 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.131747 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.131785 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.131817 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.133396 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.133476 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.133518 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.133553 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.133590 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.133621 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.134097 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.134088 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.134163 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.134163 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.134207 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.152956 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:49.157383 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:49.160025 9634 raft_consensus.cc:1203] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 07:59:49.161877 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.178663 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.191076 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:49.194811 7150 raft_consensus.cc:1203] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 07:59:49.205184 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.209973 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.211390 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:49.213379 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.213462 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.216599 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.220135 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:49.220582 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:49.275820 9634 raft_consensus.cc:1203] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 07:59:49.293468 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:49.296766 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:49.297076 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:49.297868 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:49.298265 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:49.325011 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:49.328560 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:49.377614 7282 raft_consensus.cc:1203] T fbc7505d3ef6416c9d505bfe899f73ed P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:49.393350 7282 consensus_queue.cc:237] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:49.394910 9634 raft_consensus.cc:1261] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:49.394909 7020 raft_consensus.cc:1261] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:49.395335 20737 consensus_queue.cc:1035] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:49.395627 20737 consensus_queue.cc:1035] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:49.397405 20737 raft_consensus.cc:2932] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:49.398859 9634 raft_consensus.cc:2932] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:49.401324 6742 catalog_manager.cc:5484] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 5227b26243484bbfb177a995ceb50afc reported cstate change: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:49.405568 7020 raft_consensus.cc:2932] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:49.421566 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 with delete_type TABLET_DATA_TOMBSTONED (TS 00a3f8e159c64327832d79fb32cdcca8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:49.422012 21625 tablet_replica.cc:330] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:49.422272 21625 raft_consensus.cc:2229] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:49.422546 21625 raft_consensus.cc:2258] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:49.423192 21625 ts_tablet_manager.cc:1875] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:49.426388 21625 ts_tablet_manager.cc:1888] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:49.426481 21625 log.cc:1196] T 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1ae7895f7d7f47c3ab5fb0b93c1fd8e0
I20240503 07:59:49.427127 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 1ae7895f7d7f47c3ab5fb0b93c1fd8e0 (table rebalance_test_table_13 [id=0380aa97b17745b6945aa16fc6bc7645]) successfully deleted
I20240503 07:59:49.536186 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 07:59:49.539640 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.541581 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:49.547252 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:49.552139 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:49.564265 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:49.578497 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:49.580456 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:49.635607 9765 consensus_queue.cc:237] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:49.637066 7020 raft_consensus.cc:1261] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:49.637243 7150 raft_consensus.cc:1261] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:49.637564 19823 consensus_queue.cc:1035] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:49.637835 19823 consensus_queue.cc:1035] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:49.639632 19823 raft_consensus.cc:2932] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:49.640241 7150 raft_consensus.cc:2932] T 23de5c1fe8c44b3c990f1128826a72b0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:49.643283 6745 catalog_manager.cc:5484] T 23de5c1fe8c44b3c990f1128826a72b0 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:49.644141 7020 raft_consensus.cc:2932] T 23de5c1fe8c44b3c990f1128826a72b0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cb73994515af4880b05383e78368bb4a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:49.648314 9484 tablet_service.cc:1514] Processing DeleteTablet for tablet 23de5c1fe8c44b3c990f1128826a72b0 with delete_type TABLET_DATA_TOMBSTONED (TS cb73994515af4880b05383e78368bb4a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53426
I20240503 07:59:49.649375 21686 tablet_replica.cc:330] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a: stopping tablet replica
I20240503 07:59:49.649583 21686 raft_consensus.cc:2229] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:49.649766 21686 raft_consensus.cc:2258] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:49.650707 21686 ts_tablet_manager.cc:1875] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:49.653604 21686 ts_tablet_manager.cc:1888] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:49.653748 21686 log.cc:1196] T 23de5c1fe8c44b3c990f1128826a72b0 P cb73994515af4880b05383e78368bb4a: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/23de5c1fe8c44b3c990f1128826a72b0
I20240503 07:59:49.654444 6734 catalog_manager.cc:4830] TS cb73994515af4880b05383e78368bb4a (127.0.99.134:45435): tablet 23de5c1fe8c44b3c990f1128826a72b0 (table rebalancer_extra_table_14 [id=f0fcbc958ed14af18b82b7bb244d2bfb]) successfully deleted
I20240503 07:59:49.730711 21180 raft_consensus.cc:482] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.730876 21180 raft_consensus.cc:504] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:49.731338 21180 leader_election.cc:288] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:49.731806 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa18637596d4a39aff9a4e4a261228d" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
W20240503 07:59:49.734520 6823 leader_election.cc:334] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.734613 6823 leader_election.cc:302] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92; no voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:49.734859 21180 raft_consensus.cc:2726] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:49.757099 20620 raft_consensus.cc:482] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.757246 20620 raft_consensus.cc:504] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:49.757701 20620 leader_election.cc:288] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:49.758249 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61737b6ad38c470b8b7d6fceba5dbbad" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
W20240503 07:59:49.759208 6954 leader_election.cc:334] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.759277 6954 leader_election.cc:302] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d; no voters: 3befbb2d206f4be89d3e07f38837374b, c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:49.759650 20620 raft_consensus.cc:2726] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:49.797642 7129 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:49.800130 7261 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:49.800453 9615 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:49.802250 21685 raft_consensus.cc:482] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.802398 21685 raft_consensus.cc:504] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:49.802870 21685 leader_election.cc:288] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
W20240503 07:59:49.804013 9701 leader_election.cc:334] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.815409 6868 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:49.816303 6998 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:49.816553 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866c1bf17d444315bb3eb83a7a283b54" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:49.817152 9703 leader_election.cc:302] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a; no voters: 00a3f8e159c64327832d79fb32cdcca8, 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:49.817402 21685 raft_consensus.cc:2726] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:49.823002 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:49.832885 9747 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:49.833521 21521 raft_consensus.cc:482] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.833660 21521 raft_consensus.cc:504] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:49.834131 21521 leader_election.cc:288] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:49.834429 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be911252c4da43708b7ce6592d9b17f6" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:49.834676 21521 raft_consensus.cc:482] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.834780 21521 raft_consensus.cc:504] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:49.835165 21521 leader_election.cc:288] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
I20240503 07:59:49.835917 20620 raft_consensus.cc:482] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.836012 20620 raft_consensus.cc:504] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:49.836375 20620 leader_election.cc:288] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:49.836776 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1a17fdefd3425aa0cda9a8fdc827c0" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
W20240503 07:59:49.837460 6954 leader_election.cc:334] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.837538 6954 leader_election.cc:302] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d; no voters: 00a3f8e159c64327832d79fb32cdcca8, 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:49.837777 20620 raft_consensus.cc:2726] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 07:59:49.838645 9439 leader_election.cc:334] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.838721 9439 leader_election.cc:302] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb73994515af4880b05383e78368bb4a; no voters: 3befbb2d206f4be89d3e07f38837374b, 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:49.838863 21521 raft_consensus.cc:2726] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:49.839140 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659657d85679433abdcbe711e4741b67" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
W20240503 07:59:49.840101 9439 leader_election.cc:334] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.840169 9439 leader_election.cc:302] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb73994515af4880b05383e78368bb4a; no voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:49.840585 21521 raft_consensus.cc:2726] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:49.875562 21685 raft_consensus.cc:482] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.875703 21685 raft_consensus.cc:504] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:49.876205 21685 leader_election.cc:288] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:49.877300 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25979eb7a4fe48939d0585bf27468918" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
W20240503 07:59:49.878504 9701 leader_election.cc:334] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.878576 9701 leader_election.cc:302] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a; no voters: 3befbb2d206f4be89d3e07f38837374b, 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:49.878719 21685 raft_consensus.cc:2726] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:49.886395 21521 raft_consensus.cc:482] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.886525 21521 raft_consensus.cc:504] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:49.887032 21521 leader_election.cc:288] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:49.887996 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66715134ff740129267a5300af8cb19" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
W20240503 07:59:49.888515 9439 leader_election.cc:334] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.888716 9441 leader_election.cc:302] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb73994515af4880b05383e78368bb4a; no voters: 3befbb2d206f4be89d3e07f38837374b, 5227b26243484bbfb177a995ceb50afc
I20240503 07:59:49.888871 21521 raft_consensus.cc:2726] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:49.889031 20620 raft_consensus.cc:482] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.889130 20620 raft_consensus.cc:504] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:49.889623 20620 leader_election.cc:288] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:49.890424 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be911252c4da43708b7ce6592d9b17f6" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:49.890570 9504 raft_consensus.cc:2454] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
W20240503 07:59:49.890712 6954 leader_election.cc:334] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.891005 6956 leader_election.cc:302] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:49.891232 20620 raft_consensus.cc:2781] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:49.891281 20620 raft_consensus.cc:482] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.891319 20620 raft_consensus.cc:3037] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:49.892464 20620 raft_consensus.cc:504] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:49.892812 20620 leader_election.cc:288] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:49.899714 9503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be911252c4da43708b7ce6592d9b17f6" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:49.899875 9503 raft_consensus.cc:3037] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:49.900210 21301 raft_consensus.cc:482] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.900296 21301 raft_consensus.cc:504] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:49.900723 21301 leader_election.cc:288] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:49.901158 9503 raft_consensus.cc:2454] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 2.
W20240503 07:59:49.902034 6823 leader_election.cc:334] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.903110 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61737b6ad38c470b8b7d6fceba5dbbad" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:49.903259 7020 raft_consensus.cc:2454] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:49.904487 6824 leader_election.cc:302] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:49.904685 21301 raft_consensus.cc:2781] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:49.904758 21301 raft_consensus.cc:482] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.904796 21301 raft_consensus.cc:3037] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:49.905277 6954 leader_election.cc:334] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.906131 6956 leader_election.cc:302] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, cb73994515af4880b05383e78368bb4a; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:49.906145 21301 raft_consensus.cc:504] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:49.906414 20620 raft_consensus.cc:2781] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:49.906500 20620 raft_consensus.cc:686] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:49.906615 21301 leader_election.cc:288] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:49.906646 20620 consensus_queue.cc:237] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:49.908005 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61737b6ad38c470b8b7d6fceba5dbbad" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:49.908147 7020 raft_consensus.cc:3037] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:49.909300 7020 raft_consensus.cc:2454] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 2.
W20240503 07:59:49.909785 6823 leader_election.cc:334] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.909912 6824 leader_election.cc:302] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, c4218904dd7e42d2be95b901a6e5db92; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:49.909873 6738 catalog_manager.cc:5484] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 2 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:49.911187 21301 raft_consensus.cc:2781] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:49.911284 21301 raft_consensus.cc:686] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [term 2 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:49.911435 21301 consensus_queue.cc:237] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:49.914266 6738 catalog_manager.cc:5484] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:49.964557 21301 raft_consensus.cc:482] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.964706 21301 raft_consensus.cc:504] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:49.965242 21301 leader_election.cc:288] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
W20240503 07:59:49.966450 6823 leader_election.cc:334] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.966841 21108 raft_consensus.cc:482] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.966990 21108 raft_consensus.cc:504] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:49.967433 21108 leader_election.cc:288] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
W20240503 07:59:49.968447 7085 leader_election.cc:334] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.968418 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b93171848746be89447d2ef90459a7" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:49.968938 6825 leader_election.cc:302] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92; no voters: 00a3f8e159c64327832d79fb32cdcca8, 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:49.969115 21108 raft_consensus.cc:482] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.969175 21301 raft_consensus.cc:2726] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:49.969221 21108 raft_consensus.cc:504] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:49.969657 21108 leader_election.cc:288] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:49.970082 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f55eb986bd4b7bbda10e683f3bfcad" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:49.970284 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6defd02954e009565bae98f5e8418" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:49.970654 21108 raft_consensus.cc:482] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.970750 21108 raft_consensus.cc:504] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:49.971123 7086 leader_election.cc:302] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8; no voters: 3befbb2d206f4be89d3e07f38837374b, 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:49.971457 21105 raft_consensus.cc:2726] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:49.971156 21108 leader_election.cc:288] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:49.971611 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1a17fdefd3425aa0cda9a8fdc827c0" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
W20240503 07:59:49.971649 7085 leader_election.cc:334] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.971710 7085 leader_election.cc:302] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8; no voters: 3befbb2d206f4be89d3e07f38837374b, c4218904dd7e42d2be95b901a6e5db92
I20240503 07:59:49.971735 7020 raft_consensus.cc:2454] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:49.971993 7086 leader_election.cc:302] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:49.972074 21108 raft_consensus.cc:2726] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:49.972134 21105 raft_consensus.cc:2781] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:49.972169 21105 raft_consensus.cc:482] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.972215 21105 raft_consensus.cc:3037] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:49.972537 7085 leader_election.cc:334] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.973335 21105 raft_consensus.cc:504] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:49.973629 21105 leader_election.cc:288] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383)
I20240503 07:59:49.974238 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1a17fdefd3425aa0cda9a8fdc827c0" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d"
I20240503 07:59:49.974387 7020 raft_consensus.cc:3037] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:49.974586 7085 leader_election.cc:334] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.975431 7020 raft_consensus.cc:2454] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 2.
I20240503 07:59:49.975744 7086 leader_election.cc:302] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:49.976598 21105 raft_consensus.cc:2781] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:49.976678 21105 raft_consensus.cc:686] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:49.976820 21105 consensus_queue.cc:237] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:49.979193 6738 catalog_manager.cc:5484] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:49.991762 21301 raft_consensus.cc:482] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.991896 21301 raft_consensus.cc:504] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:49.992323 21301 leader_election.cc:288] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:49.992805 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795f69249afe444ba0828875e1839b8c" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
W20240503 07:59:49.994154 6823 leader_election.cc:334] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:49.994235 6823 leader_election.cc:302] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92; no voters: 3befbb2d206f4be89d3e07f38837374b, 5227b26243484bbfb177a995ceb50afc
I20240503 07:59:49.994479 21301 raft_consensus.cc:2726] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 07:59:49.999281 19698 raft_consensus.cc:482] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:49.999402 19698 raft_consensus.cc:504] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:49.999799 19698 leader_election.cc:288] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:50.000160 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6defd02954e009565bae98f5e8418" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:50.000306 7150 raft_consensus.cc:2454] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:50.000752 6956 leader_election.cc:302] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
W20240503 07:59:50.000973 6954 leader_election.cc:334] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.001013 19698 raft_consensus.cc:2781] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.001050 19698 raft_consensus.cc:482] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.001083 19698 raft_consensus.cc:3037] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.002007 19698 raft_consensus.cc:504] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:50.002362 19698 leader_election.cc:288] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 election: Requested vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:50.002668 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6defd02954e009565bae98f5e8418" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:50.002801 7150 raft_consensus.cc:3037] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:50.003474 6954 leader_election.cc:334] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.003871 7150 raft_consensus.cc:2454] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 2.
I20240503 07:59:50.004184 6956 leader_election.cc:302] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 9c085a3241f14cb7ba1332b508f4b57d; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.004377 19698 raft_consensus.cc:2781] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.004448 19698 raft_consensus.cc:686] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:50.004602 19698 consensus_queue.cc:237] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:50.009171 6738 catalog_manager.cc:5484] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 2 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:50.009418 20737 raft_consensus.cc:482] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.009516 20737 raft_consensus.cc:504] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:50.009954 20737 leader_election.cc:288] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
W20240503 07:59:50.011757 7216 leader_election.cc:334] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.012121 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795f69249afe444ba0828875e1839b8c" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:50.012260 6886 raft_consensus.cc:2454] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 2.
I20240503 07:59:50.012480 21521 raft_consensus.cc:482] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.012593 21521 raft_consensus.cc:504] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:50.013065 21521 leader_election.cc:288] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:50.013595 7215 leader_election.cc:302] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.013824 21607 raft_consensus.cc:2781] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 07:59:50.013873 21607 raft_consensus.cc:482] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.013907 21607 raft_consensus.cc:3037] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:50.014501 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f59b9fe4a54e7d8beabce7589800b1" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:50.015189 21607 raft_consensus.cc:504] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:50.015542 21607 leader_election.cc:288] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 3 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:50.017369 21245 raft_consensus.cc:482] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.017475 21245 raft_consensus.cc:504] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:50.017675 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795f69249afe444ba0828875e1839b8c" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:50.017810 6886 raft_consensus.cc:3037] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:50.017926 21245 leader_election.cc:288] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
W20240503 07:59:50.018085 7216 leader_election.cc:334] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.018393 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa18637596d4a39aff9a4e4a261228d" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:50.018515 6888 raft_consensus.cc:2454] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:50.018909 9700 leader_election.cc:302] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:50.019160 6886 raft_consensus.cc:2454] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 3.
I20240503 07:59:50.019780 21685 raft_consensus.cc:2781] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.019845 21685 raft_consensus.cc:482] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.019888 21685 raft_consensus.cc:3037] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.020049 19698 raft_consensus.cc:482] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.020138 19698 raft_consensus.cc:504] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:50.020553 19698 leader_election.cc:288] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:50.021108 21685 raft_consensus.cc:504] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
W20240503 07:59:50.021343 9701 leader_election.cc:334] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.021481 21685 leader_election.cc:288] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 election: Requested vote from peers c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
W20240503 07:59:50.021708 6954 leader_election.cc:334] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:50.021965 9439 leader_election.cc:334] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.022039 9439 leader_election.cc:302] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb73994515af4880b05383e78368bb4a; no voters: 3befbb2d206f4be89d3e07f38837374b, a372e98a670f49e9adca46ab5c836c4a
I20240503 07:59:50.022266 21521 raft_consensus.cc:2726] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:50.022514 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25979eb7a4fe48939d0585bf27468918" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:50.022650 9765 raft_consensus.cc:2454] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:50.023154 7215 leader_election.cc:302] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, c4218904dd7e42d2be95b901a6e5db92; no voters: 3befbb2d206f4be89d3e07f38837374b
W20240503 07:59:50.023228 9701 leader_election.cc:334] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.023406 21607 raft_consensus.cc:2781] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:50.023478 21607 raft_consensus.cc:686] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [term 3 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:50.023614 21607 consensus_queue.cc:237] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:50.023972 6956 leader_election.cc:302] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.024159 20620 raft_consensus.cc:2781] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.024217 20620 raft_consensus.cc:482] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.024261 20620 raft_consensus.cc:3037] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.025372 20620 raft_consensus.cc:504] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:50.025732 20620 leader_election.cc:288] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:50.026209 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25979eb7a4fe48939d0585bf27468918" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:50.026325 9765 raft_consensus.cc:3037] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:50.026757 6954 leader_election.cc:334] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.027478 9765 raft_consensus.cc:2454] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 2.
I20240503 07:59:50.027628 6738 catalog_manager.cc:5484] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 2 to 3, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 3 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:50.027930 6956 leader_election.cc:302] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c085a3241f14cb7ba1332b508f4b57d, a372e98a670f49e9adca46ab5c836c4a; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.028421 20620 raft_consensus.cc:2781] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.028506 20620 raft_consensus.cc:686] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:50.028635 20620 consensus_queue.cc:237] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } }
I20240503 07:59:50.029557 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa18637596d4a39aff9a4e4a261228d" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:50.029691 6886 raft_consensus.cc:3037] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.030856 6886 raft_consensus.cc:2454] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 2.
I20240503 07:59:50.031749 9700 leader_election.cc:302] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, c4218904dd7e42d2be95b901a6e5db92; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.032055 21245 raft_consensus.cc:2781] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.032143 21245 raft_consensus.cc:686] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [term 2 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:50.032183 6745 catalog_manager.cc:5484] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 2 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } }
I20240503 07:59:50.032274 21245 consensus_queue.cc:237] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } }
I20240503 07:59:50.034418 20820 raft_consensus.cc:482] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.034536 20820 raft_consensus.cc:504] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:50.034932 20820 leader_election.cc:288] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:50.035199 6747 catalog_manager.cc:5484] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 2 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } }
W20240503 07:59:50.036496 9570 leader_election.cc:334] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.036731 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659657d85679433abdcbe711e4741b67" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:50.036861 9504 raft_consensus.cc:2454] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:50.037215 9572 leader_election.cc:302] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.037442 20820 raft_consensus.cc:2781] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.037494 20820 raft_consensus.cc:482] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.037528 20820 raft_consensus.cc:3037] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.038789 20820 raft_consensus.cc:504] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:50.039157 20820 leader_election.cc:288] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:50.042608 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659657d85679433abdcbe711e4741b67" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:50.042742 9504 raft_consensus.cc:3037] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.043928 9504 raft_consensus.cc:2454] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 2.
I20240503 07:59:50.045707 9572 leader_election.cc:302] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, cb73994515af4880b05383e78368bb4a; no voters:
W20240503 07:59:50.045989 9570 leader_election.cc:334] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.046545 21742 raft_consensus.cc:2781] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.046631 21742 raft_consensus.cc:686] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [term 2 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:50.046795 21742 consensus_queue.cc:237] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:50.051968 6738 catalog_manager.cc:5484] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:50.068683 21685 raft_consensus.cc:482] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.068822 21685 raft_consensus.cc:504] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:50.069264 21685 leader_election.cc:288] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:50.069743 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c23c1c8b02147b780c9e1aebc48f24a" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
W20240503 07:59:50.070366 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:50.071512 9701 leader_election.cc:334] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.071604 9701 leader_election.cc:302] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a; no voters: 00a3f8e159c64327832d79fb32cdcca8, 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.071841 21685 raft_consensus.cc:2726] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:50.111430 21105 raft_consensus.cc:482] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.111577 21105 raft_consensus.cc:504] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:50.112001 21105 leader_election.cc:288] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:50.112521 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b93171848746be89447d2ef90459a7" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:50.112687 6886 raft_consensus.cc:2454] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:50.113075 7084 leader_election.cc:302] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:50.113307 21105 raft_consensus.cc:2781] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.113360 21105 raft_consensus.cc:482] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.113396 21105 raft_consensus.cc:3037] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.114678 21105 raft_consensus.cc:504] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:50.115021 21105 leader_election.cc:288] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
W20240503 07:59:50.116756 7085 leader_election.cc:334] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.117137 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b93171848746be89447d2ef90459a7" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:50.117314 6886 raft_consensus.cc:3037] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.117867 21607 raft_consensus.cc:482] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.117964 21607 raft_consensus.cc:504] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:50.118330 21607 leader_election.cc:288] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:50.118659 6886 raft_consensus.cc:2454] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 2.
W20240503 07:59:50.119261 7085 leader_election.cc:334] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.119437 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66715134ff740129267a5300af8cb19" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:50.119570 9504 raft_consensus.cc:2454] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:50.120117 7218 leader_election.cc:302] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:50.120524 20737 raft_consensus.cc:2781] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.120580 20737 raft_consensus.cc:482] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.120616 20737 raft_consensus.cc:3037] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.121858 20737 raft_consensus.cc:504] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:50.122143 20737 leader_election.cc:288] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:50.122808 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66715134ff740129267a5300af8cb19" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:50.122943 9504 raft_consensus.cc:3037] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.124171 9504 raft_consensus.cc:2454] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 2.
I20240503 07:59:50.125315 7218 leader_election.cc:302] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:50.125835 21108 raft_consensus.cc:482] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.125953 21108 raft_consensus.cc:504] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:50.126361 21108 leader_election.cc:288] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:50.126441 21108 raft_consensus.cc:482] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.126500 21108 raft_consensus.cc:504] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:50.126808 21108 leader_election.cc:288] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:50.126765 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c23c1c8b02147b780c9e1aebc48f24a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:50.126929 9765 raft_consensus.cc:2454] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:50.127481 7087 leader_election.cc:302] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:50.127645 21807 raft_consensus.cc:2781] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.127728 21807 raft_consensus.cc:686] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [term 2 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:50.127883 21807 consensus_queue.cc:237] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:50.128239 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a4b20f1cc24ed09fdaaa2f9764e709" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
W20240503 07:59:50.128629 7085 leader_election.cc:334] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.128723 7084 leader_election.cc:302] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters: 3befbb2d206f4be89d3e07f38837374b
W20240503 07:59:50.128775 7085 leader_election.cc:334] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.129110 7087 leader_election.cc:302] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8; no voters: 3befbb2d206f4be89d3e07f38837374b, 5227b26243484bbfb177a995ceb50afc
W20240503 07:59:50.133221 7216 leader_election.cc:334] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.134934 21108 raft_consensus.cc:2781] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.135042 21108 raft_consensus.cc:482] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.135080 21108 raft_consensus.cc:3037] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.135246 6742 catalog_manager.cc:5484] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 2 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:50.136317 21108 raft_consensus.cc:504] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:50.136724 21108 leader_election.cc:288] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: Requested vote from peers a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:50.137583 21108 raft_consensus.cc:2781] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.137748 21108 raft_consensus.cc:686] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:50.138111 21108 consensus_queue.cc:237] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
W20240503 07:59:50.138304 7085 leader_election.cc:334] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.138962 21108 raft_consensus.cc:2726] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:50.139549 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c23c1c8b02147b780c9e1aebc48f24a" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:50.139704 9765 raft_consensus.cc:3037] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.141068 9765 raft_consensus.cc:2454] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 2.
I20240503 07:59:50.141364 7087 leader_election.cc:302] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters: 3befbb2d206f4be89d3e07f38837374b
W20240503 07:59:50.142956 7216 leader_election.cc:334] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.143056 21607 raft_consensus.cc:482] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.143142 21607 raft_consensus.cc:504] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:50.143571 21607 leader_election.cc:288] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130:46383), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:50.143606 6742 catalog_manager.cc:5484] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:50.143963 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3d64c714fd471b9f055af7cb172a39" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c085a3241f14cb7ba1332b508f4b57d" is_pre_election: true
I20240503 07:59:50.144760 21105 raft_consensus.cc:2781] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.144958 21105 raft_consensus.cc:686] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [term 2 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:50.145104 21105 consensus_queue.cc:237] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
W20240503 07:59:50.145576 7216 leader_election.cc:334] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.145637 7216 leader_election.cc:302] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc; no voters: 3befbb2d206f4be89d3e07f38837374b, 9c085a3241f14cb7ba1332b508f4b57d
I20240503 07:59:50.145834 21607 raft_consensus.cc:2726] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:50.147081 21607 raft_consensus.cc:482] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.147159 21607 raft_consensus.cc:504] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:50.147490 21607 leader_election.cc:288] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:50.147907 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a4b20f1cc24ed09fdaaa2f9764e709" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:50.148022 7150 raft_consensus.cc:2454] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 1.
I20240503 07:59:50.148425 7218 leader_election.cc:302] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:50.148581 21805 raft_consensus.cc:2781] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.148641 21805 raft_consensus.cc:482] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.148677 21805 raft_consensus.cc:3037] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:50.150272 7216 leader_election.cc:334] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.150424 21805 raft_consensus.cc:504] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:50.150784 21805 leader_election.cc:288] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:50.151144 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a4b20f1cc24ed09fdaaa2f9764e709" candidate_uuid: "5227b26243484bbfb177a995ceb50afc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:50.151263 7150 raft_consensus.cc:3037] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.151216 6740 catalog_manager.cc:5484] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:50.152290 21814 raft_consensus.cc:482] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.152392 7150 raft_consensus.cc:2454] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5227b26243484bbfb177a995ceb50afc in term 2.
I20240503 07:59:50.152379 21814 raft_consensus.cc:504] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:50.153026 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866c1bf17d444315bb3eb83a7a283b54" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a" is_pre_election: true
I20240503 07:59:50.153168 9765 raft_consensus.cc:2454] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
I20240503 07:59:50.153443 7218 leader_election.cc:302] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 5227b26243484bbfb177a995ceb50afc; no voters:
I20240503 07:59:50.153544 7087 leader_election.cc:302] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters:
I20240503 07:59:50.153735 21816 raft_consensus.cc:2781] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.153908 21816 raft_consensus.cc:482] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.154054 21816 raft_consensus.cc:3037] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:50.154112 7085 leader_election.cc:334] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.154273 21805 raft_consensus.cc:2781] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.154392 21805 raft_consensus.cc:686] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [term 2 LEADER]: Becoming Leader. State: Replica: 5227b26243484bbfb177a995ceb50afc, State: Running, Role: LEADER
I20240503 07:59:50.154531 21805 consensus_queue.cc:237] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:50.155835 21816 raft_consensus.cc:504] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
W20240503 07:59:50.156453 7216 leader_election.cc:334] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.157146 9765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866c1bf17d444315bb3eb83a7a283b54" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a372e98a670f49e9adca46ab5c836c4a"
I20240503 07:59:50.157104 6739 catalog_manager.cc:5484] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 5227b26243484bbfb177a995ceb50afc (127.0.99.132). New cstate: current_term: 2 leader_uuid: "5227b26243484bbfb177a995ceb50afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:50.157263 9765 raft_consensus.cc:3037] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:50.157258 7085 leader_election.cc:334] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.158416 9765 raft_consensus.cc:2454] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 2.
I20240503 07:59:50.158826 7087 leader_election.cc:302] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, a372e98a670f49e9adca46ab5c836c4a; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.158972 21814 leader_election.cc:288] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:50.159106 21814 raft_consensus.cc:2781] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.159201 21816 leader_election.cc:288] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), a372e98a670f49e9adca46ab5c836c4a (127.0.99.136:43683)
I20240503 07:59:50.159432 21814 raft_consensus.cc:686] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:50.159582 21814 consensus_queue.cc:237] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:50.166750 6739 catalog_manager.cc:5484] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:50.176204 7129 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 07:59:50.186062 6998 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:50.191614 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:50.194963 21180 raft_consensus.cc:482] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.195077 21180 raft_consensus.cc:504] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:50.195487 21180 leader_election.cc:288] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:50.196259 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f55eb986bd4b7bbda10e683f3bfcad" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:50.196400 7151 raft_consensus.cc:2454] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:50.196803 6825 leader_election.cc:302] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:50.196954 21180 raft_consensus.cc:2781] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.197019 21180 raft_consensus.cc:482] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.197055 21180 raft_consensus.cc:3037] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.198243 21180 raft_consensus.cc:504] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:50.198626 21180 leader_election.cc:288] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
W20240503 07:59:50.200306 6823 leader_election.cc:334] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:50.205367 6823 leader_election.cc:334] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.205739 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f55eb986bd4b7bbda10e683f3bfcad" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:50.205862 7150 raft_consensus.cc:3037] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.207073 7150 raft_consensus.cc:2454] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 2.
I20240503 07:59:50.208106 6825 leader_election.cc:302] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.208562 21301 raft_consensus.cc:2781] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.208648 21301 raft_consensus.cc:686] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [term 2 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:50.208775 21301 consensus_queue.cc:237] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:50.212313 9615 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:50.212797 9747 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:50.213092 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:50.215754 6739 catalog_manager.cc:5484] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:50.216639 21814 raft_consensus.cc:482] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.216754 21814 raft_consensus.cc:504] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:50.217161 21814 leader_election.cc:288] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369)
W20240503 07:59:50.219933 7085 leader_election.cc:334] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.220168 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5e8db1eec44a7881dde976e8a0a12" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:50.220595 7084 leader_election.cc:302] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8; no voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.220842 21814 raft_consensus.cc:2726] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:50.240883 21180 raft_consensus.cc:482] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.241115 21180 raft_consensus.cc:504] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:50.241684 21180 leader_election.cc:288] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:50.242306 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289b8f77bb4241bade14b1011e1de4" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
W20240503 07:59:50.243938 6823 leader_election.cc:334] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.244115 6823 leader_election.cc:302] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92; no voters: 00a3f8e159c64327832d79fb32cdcca8, 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.244354 21180 raft_consensus.cc:2726] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:50.245126 6868 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 07:59:50.248943 21814 raft_consensus.cc:482] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.249073 21814 raft_consensus.cc:504] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:50.249480 21814 leader_election.cc:288] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:50.250152 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289b8f77bb4241bade14b1011e1de4" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:50.250286 6886 raft_consensus.cc:2454] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 1.
W20240503 07:59:50.250680 7085 leader_election.cc:334] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.250828 7084 leader_election.cc:302] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.251101 21814 raft_consensus.cc:2781] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.251149 21814 raft_consensus.cc:482] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.251189 21814 raft_consensus.cc:3037] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.252372 21814 raft_consensus.cc:504] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:50.252712 21814 leader_election.cc:288] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
W20240503 07:59:50.253634 7085 leader_election.cc:334] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.253667 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289b8f77bb4241bade14b1011e1de4" candidate_uuid: "00a3f8e159c64327832d79fb32cdcca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
I20240503 07:59:50.253782 6886 raft_consensus.cc:3037] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.254868 6886 raft_consensus.cc:2454] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a3f8e159c64327832d79fb32cdcca8 in term 2.
I20240503 07:59:50.255192 7084 leader_election.cc:302] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, c4218904dd7e42d2be95b901a6e5db92; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.255441 21816 raft_consensus.cc:2781] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.255528 21816 raft_consensus.cc:686] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 LEADER]: Becoming Leader. State: Replica: 00a3f8e159c64327832d79fb32cdcca8, State: Running, Role: LEADER
I20240503 07:59:50.255661 21816 consensus_queue.cc:237] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } }
I20240503 07:59:50.261042 21741 raft_consensus.cc:482] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.261183 21741 raft_consensus.cc:504] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:50.261608 21741 leader_election.cc:288] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:50.261868 6740 catalog_manager.cc:5484] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } }
I20240503 07:59:50.262277 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5e8db1eec44a7881dde976e8a0a12" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8" is_pre_election: true
I20240503 07:59:50.262427 7150 raft_consensus.cc:2454] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:50.262718 9572 leader_election.cc:302] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2; no voters:
I20240503 07:59:50.262888 21741 raft_consensus.cc:2781] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.262955 21741 raft_consensus.cc:482] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.262997 21741 raft_consensus.cc:3037] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.264134 21741 raft_consensus.cc:504] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:50.264465 21741 leader_election.cc:288] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317)
I20240503 07:59:50.265027 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5e8db1eec44a7881dde976e8a0a12" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00a3f8e159c64327832d79fb32cdcca8"
I20240503 07:59:50.265143 7151 raft_consensus.cc:3037] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.266124 7151 raft_consensus.cc:2454] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 2.
I20240503 07:59:50.266738 9572 leader_election.cc:302] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3f8e159c64327832d79fb32cdcca8, 159177b3caeb434baa98e04136d050b2; no voters:
I20240503 07:59:50.267138 21741 raft_consensus.cc:2781] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.267220 21741 raft_consensus.cc:686] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [term 2 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:50.267699 21741 consensus_queue.cc:237] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
W20240503 07:59:50.271052 9570 leader_election.cc:334] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.271332 6740 catalog_manager.cc:5484] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
W20240503 07:59:50.271909 9570 leader_election.cc:334] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.278594 19698 raft_consensus.cc:482] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.278753 19698 raft_consensus.cc:504] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:50.279248 19698 leader_election.cc:288] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:50.279981 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3d64c714fd471b9f055af7cb172a39" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc" is_pre_election: true
I20240503 07:59:50.280113 7282 raft_consensus.cc:2454] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 1.
I20240503 07:59:50.280431 6956 leader_election.cc:302] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:50.280611 19698 raft_consensus.cc:2781] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.280656 19698 raft_consensus.cc:482] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.280689 19698 raft_consensus.cc:3037] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:50.281646 6954 leader_election.cc:334] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.281836 19698 raft_consensus.cc:504] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:50.282191 19698 leader_election.cc:288] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), 5227b26243484bbfb177a995ceb50afc (127.0.99.132:37483)
I20240503 07:59:50.282918 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3d64c714fd471b9f055af7cb172a39" candidate_uuid: "9c085a3241f14cb7ba1332b508f4b57d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5227b26243484bbfb177a995ceb50afc"
I20240503 07:59:50.283052 7282 raft_consensus.cc:3037] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.284006 21685 raft_consensus.cc:482] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.284122 21685 raft_consensus.cc:504] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:50.284304 7282 raft_consensus.cc:2454] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c085a3241f14cb7ba1332b508f4b57d in term 2.
I20240503 07:59:50.284550 21685 leader_election.cc:288] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
I20240503 07:59:50.284659 6956 leader_election.cc:302] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5227b26243484bbfb177a995ceb50afc, 9c085a3241f14cb7ba1332b508f4b57d; no voters:
I20240503 07:59:50.284843 21180 raft_consensus.cc:482] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.284934 21180 raft_consensus.cc:504] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:50.285112 9503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f59b9fe4a54e7d8beabce7589800b1" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:50.285241 9503 raft_consensus.cc:2454] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 1.
I20240503 07:59:50.285310 21180 leader_election.cc:288] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 159177b3caeb434baa98e04136d050b2 (127.0.99.135:43369), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
W20240503 07:59:50.285782 9701 leader_election.cc:334] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.285831 9634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3367cc41234d52b74eee94d6e67069" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "159177b3caeb434baa98e04136d050b2" is_pre_election: true
I20240503 07:59:50.286208 9703 leader_election.cc:302] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters: 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.286485 19698 raft_consensus.cc:2781] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.286588 19698 raft_consensus.cc:686] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 LEADER]: Becoming Leader. State: Replica: 9c085a3241f14cb7ba1332b508f4b57d, State: Running, Role: LEADER
I20240503 07:59:50.286744 19698 consensus_queue.cc:237] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } }
I20240503 07:59:50.286960 21685 raft_consensus.cc:2781] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.287017 21685 raft_consensus.cc:482] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.287051 21685 raft_consensus.cc:3037] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.288249 21685 raft_consensus.cc:504] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:50.288619 21685 leader_election.cc:288] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), cb73994515af4880b05383e78368bb4a (127.0.99.134:45435)
W20240503 07:59:50.289458 6823 leader_election.cc:334] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.289534 6823 leader_election.cc:302] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92; no voters: 159177b3caeb434baa98e04136d050b2, 3befbb2d206f4be89d3e07f38837374b
I20240503 07:59:50.289670 21301 raft_consensus.cc:2726] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:50.290418 21741 raft_consensus.cc:482] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.290405 9503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f59b9fe4a54e7d8beabce7589800b1" candidate_uuid: "a372e98a670f49e9adca46ab5c836c4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:50.290536 9503 raft_consensus.cc:3037] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.290532 21741 raft_consensus.cc:504] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
W20240503 07:59:50.291365 6954 leader_election.cc:334] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.291396 21741 leader_election.cc:288] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:50.291791 9503 raft_consensus.cc:2454] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a372e98a670f49e9adca46ab5c836c4a in term 2.
I20240503 07:59:50.292644 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3367cc41234d52b74eee94d6e67069" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:50.292807 6886 raft_consensus.cc:2454] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 1.
I20240503 07:59:50.293110 9569 leader_election.cc:302] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:50.293069 6740 catalog_manager.cc:5484] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 9c085a3241f14cb7ba1332b508f4b57d (127.0.99.130). New cstate: current_term: 2 leader_uuid: "9c085a3241f14cb7ba1332b508f4b57d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 } } }
I20240503 07:59:50.293383 21741 raft_consensus.cc:2781] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.293479 21741 raft_consensus.cc:482] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.293617 21741 raft_consensus.cc:3037] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.294705 21749 raft_consensus.cc:482] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.294801 21749 raft_consensus.cc:504] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
W20240503 07:59:50.294824 9570 leader_election.cc:334] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.295053 9703 leader_election.cc:302] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a372e98a670f49e9adca46ab5c836c4a, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:50.295205 21301 raft_consensus.cc:482] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.295289 21301 raft_consensus.cc:504] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:50.295411 21749 leader_election.cc:288] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
I20240503 07:59:50.295686 21301 leader_election.cc:288] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:50.296603 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed0aa049b8b4e7c926d6417acf4f068" candidate_uuid: "cb73994515af4880b05383e78368bb4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92" is_pre_election: true
I20240503 07:59:50.296676 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed0aa049b8b4e7c926d6417acf4f068" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a" is_pre_election: true
I20240503 07:59:50.296720 6888 raft_consensus.cc:2454] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb73994515af4880b05383e78368bb4a in term 1.
I20240503 07:59:50.296800 9504 raft_consensus.cc:2454] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 1.
I20240503 07:59:50.297066 9438 leader_election.cc:302] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:50.297325 6825 leader_election.cc:302] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:50.297464 21301 raft_consensus.cc:2781] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:50.297498 21301 raft_consensus.cc:482] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3befbb2d206f4be89d3e07f38837374b)
I20240503 07:59:50.297530 21301 raft_consensus.cc:3037] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.298622 21301 raft_consensus.cc:504] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:50.298960 21301 leader_election.cc:288] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 election: Requested vote from peers cb73994515af4880b05383e78368bb4a (127.0.99.134:45435), 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815)
I20240503 07:59:50.299679 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed0aa049b8b4e7c926d6417acf4f068" candidate_uuid: "c4218904dd7e42d2be95b901a6e5db92" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb73994515af4880b05383e78368bb4a"
I20240503 07:59:50.299788 9504 raft_consensus.cc:3037] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:50.300786 6823 leader_election.cc:334] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.300840 9504 raft_consensus.cc:2454] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4218904dd7e42d2be95b901a6e5db92 in term 2.
I20240503 07:59:50.300961 21521 raft_consensus.cc:2741] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240503 07:59:50.301209 6825 leader_election.cc:302] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4218904dd7e42d2be95b901a6e5db92, cb73994515af4880b05383e78368bb4a; no voters:
I20240503 07:59:50.301394 21301 raft_consensus.cc:2781] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.301512 21301 raft_consensus.cc:686] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [term 2 LEADER]: Becoming Leader. State: Replica: c4218904dd7e42d2be95b901a6e5db92, State: Running, Role: LEADER
I20240503 07:59:50.301635 21301 consensus_queue.cc:237] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
W20240503 07:59:50.302127 9439 leader_election.cc:334] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.302768 21245 raft_consensus.cc:2781] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.302942 21245 raft_consensus.cc:686] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [term 2 LEADER]: Becoming Leader. State: Replica: a372e98a670f49e9adca46ab5c836c4a, State: Running, Role: LEADER
I20240503 07:59:50.303079 21245 consensus_queue.cc:237] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } }
I20240503 07:59:50.305091 21741 raft_consensus.cc:504] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
I20240503 07:59:50.305485 21741 leader_election.cc:288] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 election: Requested vote from peers 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815), c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129:40655)
W20240503 07:59:50.305675 6823 leader_election.cc:334] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.305688 6740 catalog_manager.cc:5484] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to c4218904dd7e42d2be95b901a6e5db92 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:50.307773 6886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3367cc41234d52b74eee94d6e67069" candidate_uuid: "159177b3caeb434baa98e04136d050b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c4218904dd7e42d2be95b901a6e5db92"
W20240503 07:59:50.307793 9701 leader_election.cc:334] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.307896 6886 raft_consensus.cc:3037] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:50.309056 6886 raft_consensus.cc:2454] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159177b3caeb434baa98e04136d050b2 in term 2.
I20240503 07:59:50.309265 6739 catalog_manager.cc:5484] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to a372e98a670f49e9adca46ab5c836c4a (127.0.99.136). New cstate: current_term: 2 leader_uuid: "a372e98a670f49e9adca46ab5c836c4a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 } } }
I20240503 07:59:50.309388 9569 leader_election.cc:302] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159177b3caeb434baa98e04136d050b2, c4218904dd7e42d2be95b901a6e5db92; no voters:
I20240503 07:59:50.309679 21741 raft_consensus.cc:2781] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:50.309988 21741 raft_consensus.cc:686] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [term 2 LEADER]: Becoming Leader. State: Replica: 159177b3caeb434baa98e04136d050b2, State: Running, Role: LEADER
I20240503 07:59:50.310248 21741 consensus_queue.cc:237] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } }
W20240503 07:59:50.311946 9570 leader_election.cc:334] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:50.313205 6740 catalog_manager.cc:5484] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 reported cstate change: term changed from 1 to 2, leader changed from 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) to 159177b3caeb434baa98e04136d050b2 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "159177b3caeb434baa98e04136d050b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } }
I20240503 07:59:50.349771 7151 consensus_queue.cc:237] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } }
I20240503 07:59:50.351274 9765 raft_consensus.cc:1261] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:50.351336 7019 raft_consensus.cc:1261] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:50.351702 21810 consensus_queue.cc:1035] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 07:59:50.351910 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.351929 21816 consensus_queue.cc:1035] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:50.354022 9504 raft_consensus.cc:1261] T be911252c4da43708b7ce6592d9b17f6 P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:50.354115 21810 raft_consensus.cc:2932] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:50.354476 19698 consensus_queue.cc:1035] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:50.355131 7019 raft_consensus.cc:2932] T 7c2386cd009847d9a1fe7a959f47086c P 9c085a3241f14cb7ba1332b508f4b57d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:50.357497 9765 raft_consensus.cc:2932] T 7c2386cd009847d9a1fe7a959f47086c P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
I20240503 07:59:50.358211 6739 catalog_manager.cc:5484] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 reported cstate change: config changed from index 5 to 6, VOTER 3befbb2d206f4be89d3e07f38837374b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "00a3f8e159c64327832d79fb32cdcca8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 } } peers { permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "3befbb2d206f4be89d3e07f38837374b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37815 } } }
W20240503 07:59:50.366220 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:50.417279 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:50.436846 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.457221 9503 raft_consensus.cc:1261] T 659657d85679433abdcbe711e4741b67 P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:50.458122 21741 consensus_queue.cc:1035] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:50.462700 7019 raft_consensus.cc:1261] T 61737b6ad38c470b8b7d6fceba5dbbad P 9c085a3241f14cb7ba1332b508f4b57d [term 2 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:50.463441 21301 consensus_queue.cc:1035] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:50.464515 9767 raft_consensus.cc:1261] T 25979eb7a4fe48939d0585bf27468918 P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:50.465298 21811 consensus_queue.cc:1035] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:50.469007 6886 raft_consensus.cc:1261] T 795f69249afe444ba0828875e1839b8c P c4218904dd7e42d2be95b901a6e5db92 [term 3 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 07:59:50.469487 21806 consensus_queue.cc:1035] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 07:59:50.472378 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:50.480768 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.501693 7151 raft_consensus.cc:1261] T 4ef6defd02954e009565bae98f5e8418 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:50.502269 21811 consensus_queue.cc:1035] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:50.505715 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:50.519374 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:50.524796 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.531926 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:50.541204 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:50.543462 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:50.548105 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:50.549531 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:50.553184 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 07:59:50.568677 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:50.571326 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.574569 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:50.597716 7019 raft_consensus.cc:1261] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 9c085a3241f14cb7ba1332b508f4b57d [term 2 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:50.597785 7151 raft_consensus.cc:1261] T 21f55eb986bd4b7bbda10e683f3bfcad P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:50.598453 9767 raft_consensus.cc:1261] T 866c1bf17d444315bb3eb83a7a283b54 P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:50.598805 6886 raft_consensus.cc:1261] T 0aa18637596d4a39aff9a4e4a261228d P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:50.599155 21809 consensus_queue.cc:1035] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c085a3241f14cb7ba1332b508f4b57d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:50.599347 21180 consensus_queue.cc:1035] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:50.599794 21685 consensus_queue.cc:1035] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:50.600828 21816 consensus_queue.cc:1035] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:50.612401 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.645452 9503 raft_consensus.cc:1261] T d66715134ff740129267a5300af8cb19 P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:50.646040 21607 consensus_queue.cc:1035] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:50.649777 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.659440 7151 raft_consensus.cc:1261] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Refusing update from remote peer 5227b26243484bbfb177a995ceb50afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:50.659938 21607 consensus_queue.cc:1035] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:50.665189 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:50.667850 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:50.683614 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.684852 9767 raft_consensus.cc:1261] T 1c23c1c8b02147b780c9e1aebc48f24a P a372e98a670f49e9adca46ab5c836c4a [term 2 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:50.685258 21816 consensus_queue.cc:1035] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:50.698035 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.724366 9503 raft_consensus.cc:1261] T 75f59b9fe4a54e7d8beabce7589800b1 P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Refusing update from remote peer a372e98a670f49e9adca46ab5c836c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:50.725075 21245 consensus_queue.cc:1035] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:50.730248 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.745157 6886 raft_consensus.cc:1261] T 61b93171848746be89447d2ef90459a7 P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:50.745616 21816 consensus_queue.cc:1035] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:50.756263 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.785956 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:50.787672 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:50.789069 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:50.790333 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:50.801786 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:50.805445 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:50.807634 6888 raft_consensus.cc:1261] T 96289b8f77bb4241bade14b1011e1de4 P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Refusing update from remote peer 00a3f8e159c64327832d79fb32cdcca8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:50.808713 21809 consensus_queue.cc:1035] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:50.819201 6886 raft_consensus.cc:1261] T dc3367cc41234d52b74eee94d6e67069 P c4218904dd7e42d2be95b901a6e5db92 [term 2 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:50.819643 21742 consensus_queue.cc:1035] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:50.831662 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:50.838419 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.853608 7151 raft_consensus.cc:1261] T c2b5e8db1eec44a7881dde976e8a0a12 P 00a3f8e159c64327832d79fb32cdcca8 [term 2 FOLLOWER]: Refusing update from remote peer 159177b3caeb434baa98e04136d050b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:50.854058 21743 consensus_queue.cc:1035] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3f8e159c64327832d79fb32cdcca8" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:50.863047 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:50.888788 7282 raft_consensus.cc:1261] T bc3d64c714fd471b9f055af7cb172a39 P 5227b26243484bbfb177a995ceb50afc [term 2 FOLLOWER]: Refusing update from remote peer 9c085a3241f14cb7ba1332b508f4b57d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:50.889209 21812 consensus_queue.cc:1035] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b26243484bbfb177a995ceb50afc" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:50.905961 9503 raft_consensus.cc:1261] T bed0aa049b8b4e7c926d6417acf4f068 P cb73994515af4880b05383e78368bb4a [term 2 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:50.906963 21180 consensus_queue.cc:1035] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb73994515af4880b05383e78368bb4a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:50.911710 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:51.035588 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:51.036898 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:51.046494 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:51.049528 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:51.054549 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:51.070122 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:51.070928 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:51.077921 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.087633 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:51.271616 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:51.274713 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:51.280186 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:51.282817 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:51.282851 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:51.291729 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 07:59:51.322691 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:51.324066 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:51.343691 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.360788 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.365034 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:51.372906 6888 consensus_queue.cc:237] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } }
I20240503 07:59:51.374256 9767 raft_consensus.cc:1261] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:51.374295 9635 raft_consensus.cc:1261] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Refusing update from remote peer c4218904dd7e42d2be95b901a6e5db92: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:51.374663 21180 consensus_queue.cc:1035] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:51.374872 21301 consensus_queue.cc:1035] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:51.377120 21180 raft_consensus.cc:2932] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:51.378005 9767 raft_consensus.cc:2932] T 5b794508aa77492398e48d25edfb8aa3 P a372e98a670f49e9adca46ab5c836c4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:51.380615 6744 catalog_manager.cc:5484] T 5b794508aa77492398e48d25edfb8aa3 P c4218904dd7e42d2be95b901a6e5db92 reported cstate change: config changed from index 6 to 7, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "c4218904dd7e42d2be95b901a6e5db92" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
I20240503 07:59:51.382313 9635 raft_consensus.cc:2932] T 5b794508aa77492398e48d25edfb8aa3 P 159177b3caeb434baa98e04136d050b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c4218904dd7e42d2be95b901a6e5db92" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40655 } } peers { permanent_uuid: "a372e98a670f49e9adca46ab5c836c4a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43683 } } peers { permanent_uuid: "159177b3caeb434baa98e04136d050b2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43369 } } }
W20240503 07:59:51.383570 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:51.386399 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b794508aa77492398e48d25edfb8aa3 with delete_type TABLET_DATA_TOMBSTONED (TS 00a3f8e159c64327832d79fb32cdcca8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41626
I20240503 07:59:51.401782 22131 tablet_replica.cc:330] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8: stopping tablet replica
I20240503 07:59:51.402122 22131 raft_consensus.cc:2229] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:51.402509 22131 raft_consensus.cc:2258] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:51.403421 22131 ts_tablet_manager.cc:1875] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 07:59:51.403446 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:51.406069 22131 ts_tablet_manager.cc:1888] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 07:59:51.408653 22131 log.cc:1196] T 5b794508aa77492398e48d25edfb8aa3 P 00a3f8e159c64327832d79fb32cdcca8: Deleting WAL directory at /tmp/dist-test-taskZ1Z_mA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723100775159-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5b794508aa77492398e48d25edfb8aa3
I20240503 07:59:51.409487 6734 catalog_manager.cc:4830] TS 00a3f8e159c64327832d79fb32cdcca8 (127.0.99.131:36317): tablet 5b794508aa77492398e48d25edfb8aa3 (table rebalance_test_table_3 [id=5087bb8c881245579c5c955cbb80ac05]) successfully deleted
W20240503 07:59:51.415887 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.434710 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.460557 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.472671 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.487394 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:51.513155 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:51.540649 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:51.544654 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:51.547295 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:51.558211 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:51.558223 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:51.563920 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:51.575248 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:51.579936 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 07:59:51.588315 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.600720 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.605934 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.606473 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.610904 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.635450 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.684918 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.702128 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.702312 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.708704 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.716234 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.722477 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.731133 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.741382 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.744841 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:51.766590 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 07:59:51.767433 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:51.777252 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:51.777585 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:51.780689 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:51.794728 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:51.806908 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:51.822849 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:51.845041 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:51.849866 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:52.005110 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:52.007817 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:52.027773 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:52.031816 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:52.032580 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:52.034215 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:52.050987 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:52.054380 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:52.070185 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:52.229218 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:52.237517 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:52.238853 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:52.243872 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:52.262213 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:52.276513 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:52.278785 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:52.462067 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:52.476217 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:52.476408 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:52.482847 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:52.485468 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:52.488440 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:52.490571 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:52.667110 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:52.676440 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:52.689376 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:52.692286 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:52.692739 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:52.696799 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:52.702524 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:52.731896 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 07:59:52.795403 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:52.847918 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:52.883325 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:52.889922 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:52.908803 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:52.912817 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:52.922004 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:52.926688 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:52.934623 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:52.938403 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:52.940672 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 07:59:52.971585 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:52.972097 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:52.991768 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:53.016219 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:53.081037 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:53.101862 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:53.107750 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:53.110591 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:53.119331 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:53.142964 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:53.143045 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:53.143270 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 07:59:53.147457 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:53.148201 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:53.169385 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:53.170218 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:53.195070 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:53.197881 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:53.212540 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:53.213593 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:53.224907 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:53.255980 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:53.304334 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:53.381378 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:53.385334 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:53.391009 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:53.392755 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:53.395627 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:53.421396 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:53.424145 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:53.431910 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:53.607919 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:53.614648 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:53.617051 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:53.622515 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:53.623082 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:53.625931 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:53.633342 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 07:59:53.640002 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:53.651737 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 07:59:53.663064 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:53.690492 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:53.717844 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:53.737326 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:53.745859 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:53.792582 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:53.830730 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:53.838712 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:53.839990 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:53.843330 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:53.847803 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:53.855760 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:53.863360 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 07:59:53.911986 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:53.931396 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:53.934137 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:53.942147 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:53.946393 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:53.971895 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:53.989326 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:53.996991 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:54.009639 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:54.055521 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:54.057404 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:54.059860 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:54.060456 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:54.061633 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:54.069350 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:54.069365 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:54.073944 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:54.081550 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:54.086725 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:54.100790 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:54.117489 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:54.118314 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:54.119071 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:54.127990 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240503 07:59:54.130216 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:54.138221 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240503 07:59:54.139070 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:54.156144 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 07:59:54.175225 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 07:59:54.198259 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 07:59:54.198518 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240503 07:59:54.198618 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:54.280948 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:54.283329 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:54.284279 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:54.288391 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 07:59:54.290192 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:54.293836 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:54.300837 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:54.308957 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:54.323624 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:54.327054 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:54.340109 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:54.421631 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240503 07:59:54.473940 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:54.529310 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 07:59:54.533991 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:54.534581 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:54.539620 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:54.541321 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:54.546474 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:54.548839 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:54.554509 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:54.563477 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:54.760449 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 07:59:54.768079 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:54.768314 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 07:59:54.769201 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:54.773209 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:54.773638 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:54.784266 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:54.793879 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:54.794049 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:55.180672 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:55.180991 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:55.191900 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:55.202509 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:55.209075 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:55.220983 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:55.234454 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:55.268805 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.395116 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:55.402204 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:55.407181 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:55.407928 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:55.417351 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:55.430806 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:55.441118 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:55.441447 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:55.441955 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 07:59:55.452042 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:55.454681 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 07:59:55.471920 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.502027 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.514138 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.560309 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.625569 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:55.645543 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 07:59:55.645918 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:55.650224 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:55.652659 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:55.653367 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:55.655411 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:55.658483 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.660235 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:55.673611 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:55.675340 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:55.683503 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.689941 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.727561 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.795428 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.805266 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.806213 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:55.860404 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:55.865265 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:55.865592 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:55.872537 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:55.878044 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:55.879410 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 07:59:55.889832 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.890928 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:55.899452 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:55.965922 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:55.966289 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:55.996342 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:56.043148 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:56.118330 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:56.125689 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:56.128818 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 07:59:56.132737 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:56.138471 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:56.140571 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:56.141402 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:56.149406 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:56.207801 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.254227 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.291697 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:56.342927 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:56.345835 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:56.350579 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:56.350894 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 07:59:56.364667 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:56.365271 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 07:59:56.370097 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:56.376711 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:56.378528 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:56.393620 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.396134 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:56.408367 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:56.422448 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.446182 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.456967 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.463050 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.494585 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.503875 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.515763 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.551721 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.595088 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.679302 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.701447 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.723408 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:56.724679 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.769367 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.778936 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.789238 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:56.790031 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 07:59:56.795992 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:56.796767 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:56.800657 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:56.802407 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:56.805853 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:56.806514 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:56.819408 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 07:59:56.874089 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.887185 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:56.960693 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:57.001484 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:57.013711 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:57.013849 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:57.015611 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:57.021073 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:57.037959 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:57.038097 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:57.041922 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:57.044229 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:57.073717 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:57.224870 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:57.226558 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:57.249105 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:57.249711 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:57.256845 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:57.265141 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:57.265928 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:57.295334 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:57.488488 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:57.489759 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:57.494315 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:57.494916 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:57.502545 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:57.518581 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:57.520565 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 07:59:57.665649 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:57.685707 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:57.713862 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:57.718919 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:57.719621 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:57.721951 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 07:59:57.732306 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:57.750216 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:57.751950 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:57.752683 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:57.770726 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:57.772854 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:57.827466 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:57.891428 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:57.930899 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:57.938148 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:57.943496 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:57.946326 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:57.947856 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:57.967090 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:57.975381 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:57.978777 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:57.996062 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 07:59:58.078292 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:58.178898 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:58.195667 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:58.209095 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:58.210286 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:58.226042 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:58.286971 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:58.299338 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:58.307152 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:58.368614 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:58.374997 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:58.379441 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:58.384600 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:58.385083 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:58.387602 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:58.396253 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:58.403792 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 07:59:58.404254 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:58.417362 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:58.426535 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:58.460618 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:58.535136 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:58.560433 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:58.568569 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:58.569156 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:58.584506 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:58.589469 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:58.590013 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:58.590083 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:58.609148 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:58.613293 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:58.623318 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:58.625388 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:58.630036 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 07:59:58.720778 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:58.773327 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:58.787251 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 07:59:58.789213 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:58.803339 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:58.817034 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:58.818084 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:58.818408 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:58.826408 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:58.829396 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:58.829667 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:58.848915 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:58.902427 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:58.917866 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:58.934511 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:58.955469 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:58.956380 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.009755 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:59.033562 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:59.038205 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:59.042629 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 07:59:59.046981 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:59.047017 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:59.060660 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:59.062516 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:59.062579 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 07:59:59.098192 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.130254 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.140101 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 07:59:59.163204 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 07:59:59.170202 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 07:59:59.179680 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.189726 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.192250 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 07:59:59.204864 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.231385 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 07:59:59.239267 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.243731 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:59.253741 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:59.253755 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:59.264734 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:59.265372 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:59.268376 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:59.285842 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:59.294354 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.296316 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240503 07:59:59.297041 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:59.315352 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 07:59:59.320802 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.412725 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.419777 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:59.494877 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:59.495757 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:59.496157 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:59.498056 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:59.506716 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:59.527761 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:59.529083 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:59.543272 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.572187 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 07:59:59.609601 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.671495 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.783180 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:59.838691 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 07:59:59.895880 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:59.896325 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 07:59:59.897933 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 07:59:59.905164 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:59.911471 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:59.917363 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 07:59:59.932585 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 07:59:59.986295 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:00.076432 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:00.096109 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:00.100303 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:00.121721 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:00.130815 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:00.134131 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:00.135594 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:00.135880 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:00.155627 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:00.171943 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:00.224697 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:00.242370 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:00.344399 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:00.344628 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:00.349892 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:00.355741 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:00.360271 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:00.360324 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:00.360626 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:00.365561 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:00.366438 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:00.458364 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:00.461859 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:00.471381 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:00.571516 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:00.581776 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:00.584075 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:00.585189 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:00.589803 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:00.590087 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:00.621547 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:00.634413 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:00.660380 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:00.660553 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:00.687944 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:00.755188 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:00.773890 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:00.795091 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:00.799208 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:00.802830 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:00.810784 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:00.814067 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:00.822983 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:00.824584 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:00.838793 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:00.857367 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:00.887583 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:00.964344 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:01.014722 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:01.038705 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:01.041371 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:01.042871 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:01.048256 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:01.062601 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:01.070992 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:01.078049 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:01.078497 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.080159 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:01.093628 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.121719 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.129304 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:01.133865 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.159170 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.177286 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:01.180028 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.180081 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:01.207777 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.284003 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.377259 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.401072 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.446178 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.448022 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.448103 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.448766 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:01.466428 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:01.480094 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:01.480324 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:01.480590 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:01.485369 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:01.487265 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:01.506455 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:01.527729 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.604420 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.622063 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.658864 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.676573 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:01.686275 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:01.695531 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:01.696838 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:01.697392 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:01.698104 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:01.698717 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:01.699280 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:01.714318 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:01.731824 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:01.773090 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:01.797854 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.807896 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.883224 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:01.914389 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:01.916711 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:01.921519 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:01.926776 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:01.943536 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:01.962940 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:01.963543 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:01.990736 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:02.060581 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:02.104249 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:02.136977 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:02.180649 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:02.207407 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:02.209960 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:02.213738 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:02.228977 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:02.242681 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:02.253638 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:02.266906 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:02.363219 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:02.390264 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:02.440279 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:02.443997 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:02.447489 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:02.455379 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:02.456941 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:02.473351 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:02.474108 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:02.483912 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:02.497802 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:02.568177 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:02.644548 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:02.687527 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:02.694533 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:02.698518 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:02.700245 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:02.705569 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:02.705972 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:02.708966 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:02.718654 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:02.725883 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:02.792547 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:02.819975 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:02.820806 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:02.841979 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:02.931851 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:03.012022 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:03.024771 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:03.082688 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:03.101985 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:03.105607 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:03.114266 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:03.114490 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:03.116106 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:03.139509 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:03.150470 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:03.185214 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:03.207852 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:03.209122 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:03.221747 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:03.262558 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:03.276054 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:03.309051 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:03.315558 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:03.336234 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:03.345227 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:03.345894 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:03.350514 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:03.364766 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:03.365456 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:03.365756 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:03.379921 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:03.386386 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:03.399510 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:03.508428 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:03.563855 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:03.581326 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:03.584569 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:03.589555 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:03.595279 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:03.597362 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:03.608422 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:03.622812 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:03.640650 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:03.713548 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:03.728595 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:03.746619 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:03.806543 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:03.809674 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:03.809923 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:03.816921 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:03.816967 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:03.817499 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:03.817890 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:03.822290 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:03.827867 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:03.834311 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:03.854612 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:03.897955 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:03.941876 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:03.959666 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:04.036573 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:04.037269 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:04.039628 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:04.045279 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:04.048328 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:04.054884 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:04.056138 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:04.079381 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:04.082190 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:04.090106 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:04.093327 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:04.111677 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:04.168471 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:00:04.174629 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:04.193842 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:04.195636 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:00:04.206741 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:04.222577 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:04.231199 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 08:00:04.255399 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 08:00:04.261021 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:04.267590 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:04.269862 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:04.272563 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:04.279062 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:04.279891 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:04.280668 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:04.281277 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:04.299964 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:04.316161 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240503 08:00:04.317231 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:00:04.369302 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:04.376776 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:04.461129 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:04.594280 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:04.594997 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:00:04.612069 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:04.684662 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:04.700584 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:04.700677 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:04.705090 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:04.713279 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:04.714174 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:04.715121 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:04.722122 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:04.743217 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:04.756131 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:04.878145 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:04.906854 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:04.916007 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:04.916755 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:04.917723 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:04.921442 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:04.923749 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:04.926748 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:04.930119 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:05.053700 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:05.061573 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:05.112663 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:05.137537 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:05.140877 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:05.140930 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:05.144563 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:05.147639 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:05.149214 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:05.152187 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:05.165635 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:05.271114 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:05.282078 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:05.289144 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:05.343366 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:05.361649 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:05.363739 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:05.371340 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:05.372748 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:05.375087 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:05.391034 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:05.395104 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:05.493813 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:05.509828 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:05.534137 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:05.575624 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:05.597259 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:05.598731 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:05.607373 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:05.611196 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:05.611465 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:05.611874 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:05.619941 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:05.621311 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:05.650907 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:05.663653 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:05.698676 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:05.742792 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:05.759083 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:05.815124 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:05.815115 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:05.815743 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:05.826802 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:05.827273 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:05.834065 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:05.836894 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:05.864738 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:05.864837 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:05.875216 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:05.886257 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:05.935904 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:05.985780 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:06.052739 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:06.170706 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:06.223708 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:06.227106 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:06.228948 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:06.238827 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:06.245954 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.248296 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.249642 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.249727 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:06.249765 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:06.258493 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:06.278985 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:06.314409 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:06.374069 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.391489 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:06.392827 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.397933 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.415618 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.445813 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:06.447815 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:06.453218 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:06.455231 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:06.457981 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:06.464119 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:06.466539 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:06.481025 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:06.496505 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:06.524986 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.532195 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.543905 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.637256 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:06.668686 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:06.679682 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:06.682310 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:06.683437 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:06.684790 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:06.687357 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:06.693773 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:06.707999 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:06.731118 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:06.786329 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.828047 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:06.888959 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:06.889469 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:06.891029 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:06.902529 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:06.903599 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:06.917065 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:06.917065 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:06.941447 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.977247 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:06.979785 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:06.988716 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:07.026412 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:07.036492 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:07.114774 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:07.117990 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:07.118886 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:07.126926 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:07.142993 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:07.146507 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:07.162642 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:07.195103 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:07.262789 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:07.266907 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:07.330910 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:07.336019 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:07.338376 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:07.341919 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:07.351799 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:07.356985 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:07.375106 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:07.392135 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:07.423421 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:07.454805 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:07.476028 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:07.522235 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:07.758698 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:07.761632 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:07.771741 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:07.772650 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:07.775094 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:07.776607 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:07.781194 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:07.785791 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:07.796831 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:07.884022 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:07.905234 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:07.961488 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:07.961668 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:07.971563 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:07.976735 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:07.981096 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:08.001780 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:08.002104 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:08.020020 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:08.020143 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:08.032835 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:08.041322 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:08.049459 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:08.063697 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:08.092612 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:08.109854 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:08.128566 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:08.198155 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:08.199420 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:08.205904 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:08.206142 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:08.206175 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:08.208791 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:08.243245 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:08.255605 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:08.259984 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:08.357138 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:08.412395 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:08.421451 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:08.423775 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:08.426378 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:08.428304 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:08.432005 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:08.440680 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:08.456087 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:08.457446 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:08.470378 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:08.527068 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:08.527156 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:08.639814 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:08.649163 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:08.653020 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:08.654534 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:08.656184 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:08.661377 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:08.674268 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:08.677034 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:08.681542 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:08.682201 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:08.722076 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:08.725509 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:08.762507 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:08.769263 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:08.772150 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:08.804777 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:08.873142 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:08.876248 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:08.884860 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:08.888190 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:08.894800 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:08.897298 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:08.909255 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:08.911314 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:08.913969 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:08.924582 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.011848 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.020730 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.129528 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.134765 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.140858 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.142868 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.174492 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:09.189509 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.203511 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:00:09.204391 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240503 08:00:09.206102 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.211644 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.272109 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240503 08:00:09.312286 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:09.313541 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:09.315050 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:09.325697 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:09.326282 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:09.330910 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240503 08:00:09.330991 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:09.331985 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:09.374301 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:00:09.375408 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.376243 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:00:09.418537 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.484539 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.522282 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:09.536424 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:09.538538 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:09.540503 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:09.542903 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:09.545413 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:09.548207 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:09.548954 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:09.549293 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:09.569764 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.608719 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:09.637818 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:09.734974 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:09.745095 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:09.748944 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:09.755712 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:09.766012 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:09.776248 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:09.782958 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:09.809924 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:00:09.838378 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.893276 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.903373 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:09.908113 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:09.975135 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:09.980827 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:09.985769 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:09.990497 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:09.990618 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:09.993283 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:09.994282 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:10.037724 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:10.140868 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:10.147665 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:10.175923 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:10.210095 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:10.213135 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:10.213943 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:10.214671 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:10.214823 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:10.215831 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:10.231346 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:10.255964 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:10.280318 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:10.381781 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:10.423712 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:10.441865 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:10.455363 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:10.457281 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:10.462637 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:10.466548 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:10.499670 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:10.503661 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:10.508242 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:10.520977 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:10.580623 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:10.607959 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:10.619021 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:10.620005 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:10.638362 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:10.649042 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:10.694646 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:10.751530 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:10.766044 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:10.812515 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:10.879921 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:10.880369 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:10.888105 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:10.892289 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:10.893337 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:10.894815 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:10.899942 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:10.922652 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:10.945780 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:11.012770 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:11.013444 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:11.020805 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:11.068770 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:11.094722 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:11.095739 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:11.098428 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:11.105863 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:11.109895 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:11.110687 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:11.127192 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:11.128384 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:11.177088 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:11.203825 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:11.241518 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:11.330230 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:11.330485 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:11.336131 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:11.336525 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:11.340996 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:11.342836 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:11.347321 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:11.349331 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:11.365499 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:11.368618 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:11.383671 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:11.403564 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:11.410910 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:11.424582 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:11.446663 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:11.543527 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:11.552470 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:11.552667 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:11.557790 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:11.558039 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:11.566519 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:11.572124 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:11.581831 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:11.674986 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:11.681370 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:11.697800 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:11.708482 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:11.734256 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:11.759950 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:11.764148 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:11.777002 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:11.781829 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:11.784446 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:11.788995 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:11.789618 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:11.824672 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:11.834721 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:11.968617 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:11.972267 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:11.972925 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:11.980149 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:11.980415 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:11.980818 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:11.984273 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:12.001847 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:12.008788 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:12.037722 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:12.177068 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:12.208148 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:12.226438 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:12.276316 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:12.391268 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:12.397292 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:12.403476 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:12.409417 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:12.415467 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:12.432994 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:12.439805 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:12.442265 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:12.442492 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:12.446906 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:12.462504 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:12.463239 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:12.598915 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:12.627496 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:12.627904 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:12.628903 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:12.642203 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:12.644249 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:12.651232 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:12.678606 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:12.688868 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:12.746013 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:12.791815 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:12.792702 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:12.799145 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:12.846205 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:12.850144 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:12.859406 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:12.862092 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:12.869920 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:12.870653 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:12.882006 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:12.902698 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:12.987972 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:13.035199 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:13.081461 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:13.083792 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:13.083988 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:13.086117 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:13.094512 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:13.095331 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:13.095625 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:13.098414 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:13.119007 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:13.137044 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:13.137987 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:13.163692 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:13.202112 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:13.210484 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:13.232847 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:13.239135 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:13.317081 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:13.320014 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:13.320919 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:13.322700 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:13.326586 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:13.331723 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:13.332531 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:13.348244 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:13.401451 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:13.438714 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:13.469286 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:13.479419 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:13.489564 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:13.529152 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:13.554746 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:13.564410 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:13.565402 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:13.568892 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:13.574043 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:13.579227 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:13.579946 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:13.598870 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:13.615478 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:13.618392 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:13.644097 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:13.683689 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:13.749264 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:13.752867 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:13.851526 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:13.859742 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:13.945411 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:13.975466 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:13.980415 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:13.982841 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:13.990522 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:13.993588 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:14.001092 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:14.003914 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:14.014302 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:14.019929 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:14.100502 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:14.101843 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:14.106776 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:14.113026 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:14.123844 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:14.177703 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:14.187304 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:14.188992 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:14.194481 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:14.202137 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:14.204845 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:14.207903 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:14.209630 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:14.221127 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240503 08:00:14.221215 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:14.262656 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:14.268697 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:14.312667 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240503 08:00:14.382961 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:00:14.398532 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240503 08:00:14.403450 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:14.406785 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240503 08:00:14.408977 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:14.409602 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:14.420074 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:14.423328 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 08:00:14.429930 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:14.433144 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:14.434257 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:14.453723 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:14.517195 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:14.602237 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:14.626389 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:14.629274 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:14.631489 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:14.636744 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:14.641530 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:14.642007 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:14.642532 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:14.686568 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:14.758740 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:14.777977 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:14.792829 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:14.815687 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:14.848914 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:14.860109 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:14.862444 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:14.862973 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:14.874552 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:14.891803 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:14.892246 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:14.905508 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:14.967289 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.003144 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.022820 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:15.050379 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:00:15.058396 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.060025 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:15.082437 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:15.083956 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:15.087061 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:15.094770 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:15.095249 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:15.105047 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:15.109683 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:15.122622 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:15.189710 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.235785 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.263001 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.295518 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.348286 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:15.490952 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:15.501470 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:15.502777 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:15.509608 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:15.513957 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:15.523897 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:15.523994 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:15.550676 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:15.571347 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.578744 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.579839 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.609228 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:15.654523 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:15.666916 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.701650 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:15.714692 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:15.720780 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:15.722506 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:15.722779 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:15.725137 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:15.726085 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:15.749739 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:15.766059 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:15.774487 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:15.796417 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.851209 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:15.895607 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:15.899336 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:15.908289 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:15.915017 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:15.920305 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:15.925702 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:15.930702 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:15.933084 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:15.942740 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:15.976477 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:15.988422 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:16.008901 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:16.017617 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.031605 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:16.056125 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.081432 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:16.163991 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:16.173553 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:16.173972 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:16.176523 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:16.198519 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:16.198979 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:16.201560 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:16.222529 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:16.273141 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.310616 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.331655 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:16.342710 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.350383 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:16.397500 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:16.399622 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:16.406445 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:16.408720 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:16.423117 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:16.434991 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:16.438532 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:16.441054 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.506150 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.538964 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:16.541245 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.589026 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:16.608644 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:16.608656 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:16.611299 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:16.614734 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:16.618911 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:16.634546 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:16.636744 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:16.654755 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:16.711297 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.721606 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.790670 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.892974 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.932410 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.956818 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:16.963450 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:17.038031 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:17.041417 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:17.043977 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:17.049185 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:17.055655 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:17.073235 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:17.087399 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:17.108422 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:17.126472 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:17.126668 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:17.138027 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:17.147001 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:17.243098 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:17.247253 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:17.252274 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:17.260449 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:17.265777 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:17.268304 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:17.276633 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:17.285367 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:17.361948 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:17.387830 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:17.403995 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:17.469306 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:17.472524 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:17.474525 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:17.476644 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:17.493399 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:17.515692 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:17.518739 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:17.522886 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:17.528934 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:17.576015 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:17.680961 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:17.681587 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:17.685863 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:17.691092 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:17.706529 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:17.707744 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:17.718011 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:17.726897 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:17.768373 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:17.775607 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:17.850023 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:17.873848 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:17.899231 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:17.899231 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:17.906144 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:17.912525 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:17.914940 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:17.921881 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:17.935591 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:17.935570 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:17.935873 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:17.954396 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:17.956851 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:17.996348 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:17.996465 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:18.005280 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:18.066138 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:18.109088 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:18.117385 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:18.119740 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:18.120584 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:18.125058 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:18.127377 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:18.132462 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:18.136080 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:18.184494 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:18.206995 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:18.254206 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:18.254616 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:18.271454 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:18.281484 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:18.327193 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:18.329103 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:18.414017 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:18.473520 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:18.508894 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:18.557435 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:18.557435 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:18.564314 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:18.564492 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:18.565711 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:18.582204 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:18.591475 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:18.613911 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:18.623888 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:18.734324 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:18.749219 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:18.775764 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:18.779646 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:18.781599 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:18.782773 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:18.787987 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:18.788087 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:18.790500 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:18.791880 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:18.826995 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:18.882936 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:18.920611 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:18.951045 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:18.957862 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:18.978708 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:18.979599 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:18.986640 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:18.995887 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:18.998067 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:19.002363 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:19.031472 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:19.073302 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:19.103547 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:19.135684 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:19.142000 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:19.200547 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:19.211385 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:19.211905 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:19.224812 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:19.227543 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:19.255756 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:19.261855 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:19.314877 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:00:19.317703 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240503 08:00:19.326694 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:19.355269 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:19.371568 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:19.393527 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:19.418902 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:19.422840 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240503 08:00:19.428258 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:19.432938 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:19.434008 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:19.448501 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:19.448585 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:19.452623 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:19.452772 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:19.458211 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240503 08:00:19.489874 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:19.495390 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 08:00:19.562222 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:00:19.583745 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:19.592931 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:19.596275 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:19.600935 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:19.631913 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:19.657141 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:19.663161 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:19.665416 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:19.680984 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:19.689555 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:19.692889 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:19.702215 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:19.709443 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:19.720324 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:19.781837 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:19.980144 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:20.031220 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:20.064638 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:20.094823 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:00:20.095559 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:20.110954 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:20.113716 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:20.117333 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:20.118772 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:20.131928 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:20.132040 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:20.139528 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:20.150751 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:20.150947 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:20.246060 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:20.268024 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:20.279896 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:20.307068 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:20.323734 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:20.330205 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:20.333333 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:20.334579 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:20.337541 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:20.338716 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:20.348182 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:20.371748 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:20.390765 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:20.422848 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:20.439340 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:20.502594 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:20.516839 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:20.535396 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:20.547912 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:20.571522 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:20.575012 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:20.576706 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:20.579478 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:20.580860 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:20.581624 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:20.582513 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:20.583905 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:20.638274 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:20.660926 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:20.664522 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:20.684648 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:20.704804 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:20.782021 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:20.792281 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:20.793821 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:20.797981 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:20.803457 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:20.804782 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:20.807852 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:20.852459 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:20.859576 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:20.955780 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:20.981707 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:20.990707 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:20.998649 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:20.999269 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:21.005424 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:21.027397 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:21.029790 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:21.042935 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:21.122654 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:21.122815 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:21.158627 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:21.172173 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:21.190490 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:21.218358 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:21.223574 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:21.232326 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:21.232661 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:21.238507 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:21.254753 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:21.274080 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:21.329061 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:21.335974 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:21.340610 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:21.361718 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:21.376205 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:21.420991 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:21.428458 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:21.498826 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:21.637472 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:21.640244 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:21.646622 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:21.647789 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:21.648713 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:21.665500 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:21.667052 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:21.679193 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:21.686594 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:21.694597 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:21.696146 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:21.736766 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:21.737298 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:21.772156 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:21.842517 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:21.845254 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:21.848229 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:21.850214 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:21.852972 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:21.853410 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:21.854851 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:21.868176 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:21.899454 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:22.046988 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:22.049118 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:22.051919 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:22.053540 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:22.054109 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:22.075001 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:22.078541 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:22.079231 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:22.096995 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:22.118072 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:22.123361 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:22.126328 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:22.155058 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:22.161788 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:22.206472 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:22.277257 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:22.278479 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:22.279767 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:22.282436 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:22.286597 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:22.294795 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:22.299420 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:22.316592 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:22.462245 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:22.494010 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:22.497962 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:22.504662 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:22.512655 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:22.512933 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:22.524641 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:22.533409 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:22.544965 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:22.583567 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:22.681710 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:22.708798 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:22.717468 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:22.717511 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:22.720185 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:22.722623 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:22.723260 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:22.726114 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:22.768098 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:22.768074 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:22.882570 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:22.890328 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:22.959405 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:22.966971 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:22.972326 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:23.022660 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:23.073621 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:23.104419 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:23.105062 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:23.128152 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:23.133905 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:23.137848 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:23.138815 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:23.139289 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:23.139403 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:23.159551 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:23.162468 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:23.164477 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:23.165267 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:23.167601 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:23.171450 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:23.177263 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:23.219487 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:23.305150 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:23.346768 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:23.362562 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:23.363816 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:23.364077 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:23.365013 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:23.366545 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:23.367978 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:23.367985 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:23.376927 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:23.384311 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:23.433720 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:23.450593 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:23.536424 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:23.557171 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:23.564519 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:23.581533 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:23.585501 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:23.589310 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:23.594790 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:23.609338 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:23.610101 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:23.625056 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:23.638237 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:23.652721 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:23.686209 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:23.735599 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:23.789516 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:23.802098 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:23.804841 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:23.812956 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:23.821640 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:23.824541 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:23.825038 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:23.831295 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:23.837988 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:23.850497 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:23.990024 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:24.005178 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:24.009407 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:24.020000 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:24.029407 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:24.029428 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:24.035138 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:24.039197 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:24.046007 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:24.060977 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:24.116832 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:24.131356 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:24.153702 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:24.175503 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:24.230589 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:24.233184 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:24.235073 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:24.244048 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:24.262058 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:24.262596 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:24.262966 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:24.285431 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:24.333700 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:24.336159 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 08:00:24.425977 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:24.442931 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240503 08:00:24.456279 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:24.493149 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:00:24.506210 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:24.516176 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240503 08:00:24.516894 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:24.558069 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:24.567572 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 08:00:24.568387 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:24.578025 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:00:24.627224 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:24.634472 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:24.690948 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:24.693630 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:24.696125 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:24.696415 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:24.702512 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:24.705257 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:24.706516 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:24.729763 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:24.740473 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:24.913800 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:24.914532 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:24.915107 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:24.922310 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:24.924027 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:24.943661 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:24.946985 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:24.954461 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:24.975018 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:25.051023 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:25.097318 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:00:25.130000 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:25.138289 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:25.138703 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:25.142149 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:25.143553 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:25.144150 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:25.172235 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:25.182631 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:25.206246 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:25.206924 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:25.250628 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:25.315932 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:25.352962 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:25.360606 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:25.361912 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:25.362870 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:25.367868 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:25.371843 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:25.375484 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:25.436965 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:25.478863 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:25.537678 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:25.539115 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:25.545393 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:25.557510 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:25.572543 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:25.578711 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:25.583448 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:25.584903 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:25.588050 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:25.588165 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:25.592164 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:25.603603 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:25.622609 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:25.632745 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:25.640081 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:25.658691 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:25.667969 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:25.678648 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:25.722276 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:25.776252 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:25.799702 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:25.805617 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:25.809808 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:25.812220 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:25.812229 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:25.812870 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:25.850545 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:25.918491 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:25.919826 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:25.952227 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:25.971694 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:25.980162 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:25.989576 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:26.035406 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:26.056598 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:26.154601 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:26.188820 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:26.193542 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:26.202965 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:26.222000 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:26.232754 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:26.247987 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:26.250224 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:26.258136 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:26.259032 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:26.266669 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:26.274062 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:26.295835 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:26.300854 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:26.351666 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:26.400967 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:26.452153 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:26.452697 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:26.463053 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:26.464538 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:26.470674 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:26.479244 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:26.490005 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:26.498392 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:26.518227 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:26.599854 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:26.612183 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:26.665822 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:26.667822 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:26.668011 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:26.681962 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:26.687027 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:26.687532 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:26.702405 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:26.708343 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:26.726469 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:26.811985 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:26.849349 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:26.878306 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:26.878819 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:26.881780 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:26.886611 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:26.898419 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:26.899227 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:26.941510 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:26.942219 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:26.980121 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:27.009965 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:27.057466 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:27.098443 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:27.100775 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:27.112583 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:27.112722 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:27.114109 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:27.117770 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:27.136379 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:27.152997 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:27.258530 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:27.267906 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:27.268870 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:27.272490 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:27.306224 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:27.315948 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:27.322805 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:27.323179 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:27.327553 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:27.337584 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:27.343899 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:27.346251 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:27.432492 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:27.452865 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:27.534060 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:27.548712 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:27.625280 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:27.657816 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:27.668388 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:27.687232 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:27.733034 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:27.733786 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:27.738526 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:00:27.754941 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:27.755275 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:27.761464 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:27.764348 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:27.766906 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:27.883101 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:27.942546 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:27.952649 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:27.958375 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:27.962512 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:27.963846 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:27.965152 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:27.970331 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:27.976222 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:27.992416 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:28.004786 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:28.018368 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:28.051537 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:28.101114 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:28.106925 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:28.164438 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:28.170657 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:28.175949 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:28.180905 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:28.183601 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:28.184134 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:28.190737 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:28.194941 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:28.244891 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:28.265851 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.272192 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.285898 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:28.286892 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:28.302063 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.346817 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:28.399933 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:28.403379 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:28.404417 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:28.405019 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:28.412102 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:28.434657 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:28.438822 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:28.439183 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.451725 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:28.490897 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:28.540928 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.559139 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.616847 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:00:28.624357 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:28.627408 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:28.630103 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:28.646484 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:28.648532 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:28.652931 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:28.670861 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:28.708134 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.734159 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.734265 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.746398 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:28.778301 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.778918 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.785742 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.788379 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:28.852794 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:28.854676 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:28.856276 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:28.856935 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:28.858494 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:28.861047 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:28.874179 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:28.879827 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:00:28.891551 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:28.944962 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:00:28.946079 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:28.969079 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:29.133639 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:00:29.252806 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:29.261351 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:29.263298 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:29.264629 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:29.283102 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:00:29.288810 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:29.293571 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:29.317631 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:29.336988 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 08:00:29.448154 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 08:00:29.454545 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:29.455452 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:00:29.490051 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:29.492904 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:29.493443 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:29.498667 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:29.504678 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:29.512893 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240503 08:00:29.514120 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:29.519147 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240503 08:00:29.524293 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:00:29.527096 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:29.535252 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:29.553679 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:29.619143 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240503 08:00:29.620121 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:29.675627 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:00:29.676867 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:00:29.711267 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:29.720711 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:29.720870 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:29.721220 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:29.723373 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:29.729840 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:29.743081 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:29.747189 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:29.780139 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:29.814786 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:29.841547 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:29.867342 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:29.909812 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:29.927451 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:29.931054 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:29.933282 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:29.939080 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:29.943291 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:29.953562 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:29.956954 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:29.958137 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:00:29.962069 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:29.993252 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:29.997695 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:30.049257 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:30.152050 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:30.152594 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:30.154973 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:30.173696 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:30.174177 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:30.179821 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:30.180882 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:30.223150 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:00:30.223906 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:30.236560 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:30.299679 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:30.372864 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:00:30.379418 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:30.386242 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:00:30.387111 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:30.390003 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:30.392788 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:30.401623 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:30.403645 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:30.437994 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:30.449954 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:30.468880 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:30.491706 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:30.507350 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:30.524919 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:30.575042 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:30.755497 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:30.764729 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:30.782223 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:30.806186 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:00:30.832492 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:30.836200 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:30.840379 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:30.842502 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:30.844141 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:30.848963 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:30.855628 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:30.860783 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:30.870918 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:30.894481 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:30.902549 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:30.932292 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:30.983919 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:30.999828 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:31.015218 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:31.019134 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:31.043550 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:31.046838 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:31.049993 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:31.053086 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:31.053843 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:31.075663 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:31.078948 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:31.078948 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:31.090701 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:31.091691 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:31.095666 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:31.108186 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:31.115609 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:31.137593 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:31.156726 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:31.207576 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:31.255859 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:31.265200 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:31.268972 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:31.281378 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:31.284140 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:31.286530 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:31.292819 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:31.296705 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:31.319306 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:31.489897 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:31.490495 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:31.492851 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:31.501317 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:31.502136 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:31.514897 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:31.533545 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:31.536015 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:31.627352 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:00:31.709426 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:31.715284 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:31.719484 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:31.720614 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:31.721947 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:31.723949 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:31.737301 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:31.766544 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:31.804560 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:31.881639 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:31.938913 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:00:31.975244 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:31.994204 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:32.005795 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:32.016523 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:32.031263 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:32.038560 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:32.046451 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:32.051859 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:32.052296 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.079120 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:32.082741 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.118863 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.185765 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.370800 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:32.370918 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.383396 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.383844 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.385718 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:32.441365 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:32.445852 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:32.447701 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:32.453905 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:32.458573 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:32.466372 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:32.472134 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:32.487133 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.500236 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:32.504710 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:32.518724 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:32.527549 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.539728 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.628306 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:32.660425 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:32.666558 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:32.668397 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:32.668697 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:32.672363 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:32.702515 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:32.703437 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:32.716823 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:32.735954 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.741598 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.876255 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.892380 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:32.901264 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:32.902544 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:32.902653 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:32.907629 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:32.910197 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:32.910742 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:32.939000 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:32.971433 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:32.978152 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:33.012785 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:33.058158 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:33.110459 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:33.127105 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:33.131055 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:33.135150 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:33.137126 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:33.137849 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:33.147269 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:33.166911 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:33.195379 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:33.312072 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:33.348749 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:33.351241 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:33.353099 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:33.355573 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:33.361078 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:33.364213 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:33.378540 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:33.395267 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:33.401255 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:33.404160 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:33.404770 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:33.411557 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:33.446333 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:33.458477 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:33.462222 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:33.469396 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:33.480207 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:33.508247 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:33.540524 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:33.544106 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:33.557034 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:33.583443 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:33.584277 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:33.589278 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:33.593019 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:33.598596 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:33.601111 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:33.606444 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:33.606663 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:33.606928 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:33.607245 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:33.608120 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:33.610890 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:33.702651 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:33.724057 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:33.804801 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:33.998093 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:34.008522 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:34.009943 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:34.018127 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:34.018127 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:34.031071 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:34.031865 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:34.073836 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:34.200743 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:34.239250 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:34.239531 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:34.246646 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:34.257328 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:34.266628 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:34.280493 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:34.293421 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:34.303311 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:34.327720 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:34.380434 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 08:00:34.383479 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:34.464702 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:00:34.475765 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:34.479426 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240503 08:00:34.489816 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:34.492817 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:34.493223 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:34.496574 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:34.505872 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:34.505898 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:34.511435 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:00:34.537317 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 08:00:34.550561 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:34.588186 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:00:34.595353 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:34.646744 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:34.649742 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:34.677924 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:00:34.679127 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:34.697892 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:34.699115 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:34.701853 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:34.702128 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:34.703047 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:34.707293 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:34.715831 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:34.720383 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:34.779847 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:00:34.817857 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:34.852284 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:34.856480 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:34.858932 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:34.906520 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:34.906852 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:34.909595 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:34.915434 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:34.915689 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:34.916910 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:34.922729 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:34.942515 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:34.953979 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:34.964244 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.004492 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.048274 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:35.085701 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:35.122777 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:35.128623 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:35.142961 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:35.143791 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:35.152669 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:35.154294 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:35.157655 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:35.176487 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:35.199118 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.227598 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:00:35.310742 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.326515 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.330493 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:35.334301 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.406185 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.497287 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.520995 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:35.545019 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:35.546082 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:35.561748 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:35.563344 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:35.563632 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:35.579319 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:35.579792 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:35.582880 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:35.583416 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:35.630278 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.637115 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.725200 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:35.758817 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:35.763695 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:35.765123 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:35.765393 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:35.766960 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:35.777954 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:35.789692 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:35.791451 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:35.795516 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.798480 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:35.828148 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.844648 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.854727 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:35.925091 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:35.939774 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:35.967459 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:35.972718 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:35.975126 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:35.976138 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:35.998688 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:36.004024 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:36.007383 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:36.010740 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:36.014561 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:36.028168 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:36.035058 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:36.036748 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:36.039788 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:36.065186 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:36.110141 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:36.136014 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:36.216327 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:36.217249 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:36.229352 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:36.230037 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:36.230790 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:36.231547 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:36.248322 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:36.256928 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:36.257658 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:36.363005 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:36.374269 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:36.446641 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:36.457306 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:36.458285 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:36.471128 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:36.474457 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:36.478506 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:36.480548 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:36.511004 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:36.656432 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:36.686525 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:36.692353 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:36.694766 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:36.695632 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:36.696761 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:36.699983 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:36.711807 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:36.786217 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:36.814239 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:36.819178 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:36.929584 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:36.974047 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:36.982362 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:36.993503 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:37.076428 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:00:37.101346 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:37.101850 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:37.103417 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:37.114377 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:37.114928 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:37.116868 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:37.120179 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:37.121500 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:37.121835 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:00:37.157464 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:37.162153 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:37.182660 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:37.225371 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:37.275197 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:37.309422 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:37.314179 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:37.316949 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:37.318528 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:37.321718 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:37.338541 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:37.343755 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:37.396018 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:37.451176 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:37.532550 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:37.533027 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:37.533207 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:37.539855 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:37.550633 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:37.555059 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:37.557399 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:37.557492 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:37.582190 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:37.587613 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:37.637319 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:37.646242 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:37.678220 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:37.716794 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:37.743896 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:37.745306 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:37.749967 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:37.757547 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:37.761059 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:37.763751 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:37.772511 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:37.778952 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:37.860198 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:37.893981 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:37.915885 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:37.963158 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:37.968884 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:37.973055 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:37.973608 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:37.985193 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:37.987601 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:37.999527 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:38.001016 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:38.037544 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:38.041854 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:38.054613 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:38.086241 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:38.111688 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:38.124018 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:38.147797 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:38.159921 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:38.187647 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:38.196997 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:38.206321 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:38.215926 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:38.221959 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:38.221956 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:38.230763 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:38.244714 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:38.258786 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:38.266808 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:38.279875 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:38.288002 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:38.311497 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:38.412722 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:38.486011 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:38.523870 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:38.599252 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:38.614485 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:00:38.621691 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:38.623342 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:38.632251 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:38.647536 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:38.648137 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:38.655433 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:38.668512 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:38.678625 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:38.683154 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:38.690869 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:38.698328 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:38.712045 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:38.718596 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:38.831293 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:38.837479 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:38.846668 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:38.846765 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:38.856423 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:38.871065 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:38.876346 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:39.004743 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:39.056386 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:39.064525 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:39.065038 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:39.065054 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:39.094532 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:39.094810 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:39.106518 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:39.138753 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:39.217705 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:39.220816 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:39.239058 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:39.285770 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:39.290817 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:39.291173 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:39.293751 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:39.312105 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:39.327040 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:39.327328 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:39.347658 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:39.400501 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:39.407398 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:39.447860 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:39.451804 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:39.473368 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 102 similar messages]
I20240503 08:00:39.512533 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:39.515313 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:39.516814 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:39.519240 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:39.519642 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:39.521150 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:39.534103 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:39.535346 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 08:00:39.537612 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240503 08:00:39.541473 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:39.542773 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:39.582607 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:39.588567 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:39.590158 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:39.625221 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240503 08:00:39.692832 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:00:39.708784 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:39.718271 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:39.726151 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:39.732295 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:39.738037 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:39.742770 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:39.754935 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:39.769907 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:39.770619 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:39.779573 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:39.860389 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:39.871337 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:00:39.900272 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:39.920096 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:00:40.012112 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:40.056385 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:40.141244 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:40.149387 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:40.151381 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:40.156092 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:40.156612 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:40.158638 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:40.163746 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:40.166654 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:40.179163 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:40.188027 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:40.242950 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:40.308216 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:40.344538 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:40.345443 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:40.360061 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:40.368541 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:40.368858 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:40.377125 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:40.380748 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:40.386029 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:40.394138 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:40.397118 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:40.399394 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:40.431408 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:40.480350 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:00:40.481427 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:40.501781 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:40.502228 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:40.521298 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:40.600580 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:40.603721 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:40.608186 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:40.614701 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:40.616662 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:40.628314 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:40.632383 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:40.633553 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:40.644207 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:40.650162 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:40.656193 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:40.682026 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:40.706779 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:40.739620 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:40.757061 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:40.764262 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:40.833195 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:40.834841 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:40.837209 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:40.842420 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:40.849015 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:40.856793 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:40.875834 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:40.879209 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:40.880625 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:40.967940 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:40.970829 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:41.031592 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:41.070943 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:41.074664 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:41.075667 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:41.085903 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:41.090548 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:41.097429 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:41.107704 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:00:41.111327 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:41.122435 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:41.149348 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:41.163280 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:41.256629 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:41.296216 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:41.299890 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:41.303544 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:41.306782 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:41.322999 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:41.334020 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:41.334941 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:41.337621 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:41.337836 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:41.351073 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:41.501107 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:41.502432 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:41.567332 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:41.704308 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:41.711091 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:41.713459 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:41.717458 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:41.723006 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:41.724321 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:41.724882 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:41.744417 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:41.866135 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:41.874909 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:41.885695 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:41.899385 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:41.914583 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:41.915477 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:41.919737 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:41.927587 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:41.927884 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:41.928267 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:41.951840 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:41.974136 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.056740 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.076181 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.138396 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:42.139515 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:42.144281 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:42.146530 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:42.155089 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:42.155954 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:42.168507 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:42.181571 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:42.192698 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.340740 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:42.362316 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:42.366976 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:42.377882 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:42.378844 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:42.388761 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.394369 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:00:42.394996 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:42.398535 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:42.400800 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:42.401896 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:42.414760 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:42.530104 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.544598 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.581100 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:42.595760 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:42.597872 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:42.611166 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:42.615796 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:42.622740 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:42.623848 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:42.624758 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:42.636783 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.641781 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.676093 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.729681 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:42.761569 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:42.825416 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:42.832314 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:42.834558 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:42.835641 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:42.842506 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:42.843268 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:42.854662 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:42.857762 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:42.893497 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:42.905151 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.931252 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.940169 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:42.943234 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:43.013816 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:43.058915 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:43.060022 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:43.072947 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:43.084048 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:43.089533 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:43.090179 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:43.090222 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:43.108414 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:43.217288 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:43.234613 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:00:43.243084 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:43.243922 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:43.247267 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:43.249902 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:43.250574 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:43.258450 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:43.271417 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:43.278939 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:43.337671 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:43.408655 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:43.435581 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:43.447644 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:43.450994 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:43.461539 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:43.462849 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:43.464007 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:43.473949 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:43.488009 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:43.504774 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:43.543154 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:43.555508 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:43.609051 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:43.622826 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:43.667892 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:43.678840 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:43.679350 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:43.681869 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:43.690696 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:43.698736 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:43.702450 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:43.714563 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:43.717155 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:43.774001 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:43.832568 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:43.846765 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:00:43.879235 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:43.884253 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:43.887924 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:43.896351 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:43.902866 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:43.902913 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:00:43.907121 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:43.911417 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:43.924774 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:44.061328 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:44.102466 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:44.108243 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:44.116127 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:44.116544 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:44.117440 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:44.117971 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:44.138532 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:44.147931 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:44.179500 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:44.211876 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:44.244724 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:00:44.325233 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:44.329336 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:44.333187 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:44.336736 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:00:44.338608 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:44.350129 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:44.354784 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:44.356707 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:44.382200 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:00:44.494665 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:44.501708 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:44.509662 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 08:00:44.574193 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 08:00:44.575397 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:44.583199 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:44.592813 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:44.620958 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:44.623440 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240503 08:00:44.701345 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:00:44.749150 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:44.757464 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:44.759683 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:44.763610 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:44.768497 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:44.772595 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:00:44.774819 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:44.793040 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 103 similar messages]
I20240503 08:00:44.801314 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:44.852108 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:00:44.859599 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:44.872666 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:44.905627 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:44.915426 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:00:44.957001 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:44.957139 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:44.967152 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:44.967360 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:44.976399 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:00:44.992615 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:45.005923 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:45.014027 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:45.025851 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.072139 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:45.108655 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:45.171852 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:45.178779 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:45.182595 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:45.184743 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:45.197590 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:45.203112 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:00:45.203202 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:45.206894 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:45.232854 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.297312 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.299939 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.302647 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:45.331460 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.338238 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:00:45.396183 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:45.396319 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:45.404420 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:45.415505 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:45.415923 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:45.432548 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:45.441643 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.441751 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:45.454150 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:45.483490 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.497164 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:45.510244 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.528403 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:45.544420 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:00:45.545835 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.550264 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.583388 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:45.616371 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:45.621028 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:45.626097 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:45.627734 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:45.637712 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:45.641767 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:45.643335 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:45.659039 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:45.659916 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.799695 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.825814 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:00:45.856498 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:45.859828 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:45.861069 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:45.867193 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:45.875092 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:45.877264 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:45.889451 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:45.898510 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:45.899432 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:45.917663 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.944396 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:45.951676 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:45.966967 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:46.087378 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:46.105296 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:46.159103 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:46.177803 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:46.187569 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:46.198165 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:46.234982 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:46.276546 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:46.284818 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:46.292791 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:46.295430 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:46.300518 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:46.310117 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:46.318037 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:46.322809 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:46.353912 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:46.444175 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:46.519043 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:46.521080 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:46.522259 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:46.538421 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:46.539305 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:46.555581 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:46.573086 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:46.618840 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:46.685577 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:46.742151 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:46.746551 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:46.759845 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:00:46.763628 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:46.764588 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:46.773686 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:46.774232 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:46.783596 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:46.789548 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:46.803061 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:46.847681 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:46.896858 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:46.905182 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:46.914605 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:46.957902 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:46.958106 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:46.970757 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:46.980372 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:46.980618 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:46.981364 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:46.984560 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:47.097158 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:47.103919 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.105262 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.108228 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:47.174013 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:47.176194 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:47.180991 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:47.181241 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:47.182493 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:47.182955 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:47.220680 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:47.229852 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:47.245913 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.294935 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:00:47.390651 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:47.399003 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:47.401816 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:47.402506 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:47.404315 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:47.412720 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:47.414084 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:47.431176 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:47.435699 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:47.516171 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.522166 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:47.567514 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.587447 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.631402 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:47.637475 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.644011 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.651800 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.660511 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.676190 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:47.686555 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.747982 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:47.820050 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:47.820520 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:47.825915 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:47.828894 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:47.850807 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:47.859619 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:47.864259 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:47.876822 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.896096 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:47.933211 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:47.980500 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:47.989809 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:47.998252 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:48.034446 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:48.041661 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:48.050705 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:48.051002 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:48.069783 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:48.076691 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:48.096254 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:48.137938 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:48.242929 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:48.243815 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:48.267143 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:48.267143 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:48.271651 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:48.276556 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:48.279143 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:48.279412 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:48.280969 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:48.289191 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:48.292698 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:48.300737 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:48.314463 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:48.409154 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:48.468959 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:48.476626 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:48.491528 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:48.497705 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:48.499213 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:48.507359 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:48.507694 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:48.509038 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:48.516377 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:48.538508 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:48.554698 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:48.583787 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:48.603217 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:48.639187 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:48.683248 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:48.714663 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:48.721870 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:48.723145 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:48.724305 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:48.726927 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:48.733276 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:48.738302 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:48.739771 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:48.776939 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:48.847662 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:48.849360 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:48.915453 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:48.921882 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:48.927487 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:48.931331 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:48.932086 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:48.933882 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:48.943773 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:48.950229 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:48.972658 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:49.001782 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:49.171720 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:49.282773 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:49.313125 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:49.314492 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:49.324155 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:00:49.357968 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:49.362176 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:49.366523 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:49.366765 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:49.373909 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:49.386899 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:49.387894 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:49.391798 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:49.451758 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:49.464507 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:49.479468 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:49.480657 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:49.535629 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:49.547583 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:00:49.568823 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:49.569089 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:49.571741 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:49.578816 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:49.583252 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:49.583626 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:49.584203 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:49.600530 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240503 08:00:49.600847 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:49.605470 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:49.643362 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240503 08:00:49.656286 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:49.784057 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:49.788758 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:49.790961 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:49.800808 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:49.802122 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:49.815972 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:49.822837 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:49.832126 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:49.848124 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240503 08:00:49.849702 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:00:49.969583 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:49.975308 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240503 08:00:49.995784 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:49.998951 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:50.002535 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:50.013039 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:50.013430 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:50.020457 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:50.029966 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:50.035945 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:50.039472 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:50.056964 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.100407 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:00:50.131253 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.134873 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.160836 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.175248 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:50.231361 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:50.235173 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:50.239070 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:50.241255 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:50.241537 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:50.250015 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.266953 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.268242 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:50.271719 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:50.271989 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:50.278929 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:50.287600 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.438624 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:50.451612 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:50.461014 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:50.461500 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:50.465412 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:50.466892 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:50.476480 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:50.477227 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:50.492262 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:50.495625 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:50.521030 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:50.573300 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:50.594856 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:00:50.620805 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.626804 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:50.637945 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:50.705726 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.781476 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.813048 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.816231 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:50.827944 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:50.856267 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.859925 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:50.880582 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:50.881517 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:50.885823 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:50.892498 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:50.899505 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:50.900457 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:50.906486 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:50.958281 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:50.958765 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:51.048905 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:51.049091 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:51.084794 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:51.094503 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:51.095451 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:51.095729 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:51.097503 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:51.107545 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:51.131876 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:51.208465 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:51.209300 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:51.243785 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:51.256037 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:51.256775 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:51.275516 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:51.320731 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:51.329532 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:51.330065 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:51.330202 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:51.331959 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:51.335526 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:51.348196 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:51.361079 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:51.541277 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:51.548087 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:51.550540 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:51.550843 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:51.553292 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:51.558751 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:51.563267 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:51.571316 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:00:51.579149 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:51.586447 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:51.620127 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:51.661804 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:51.677618 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:51.703562 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:51.756584 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:51.756767 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:51.772130 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:51.780622 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:51.785890 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:51.799903 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:51.800406 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:51.808243 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:51.914031 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:51.929397 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:51.963788 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:51.979430 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:51.983675 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:51.989639 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:51.995591 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:51.997157 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:52.011067 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:52.022084 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:52.022446 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:52.025285 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:52.038269 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:52.125072 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:52.135108 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:52.248844 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:52.282272 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:52.360129 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:52.395395 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:52.402489 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:52.406134 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:52.406702 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:52.410923 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:52.430022 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:52.430919 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:52.455190 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:52.531505 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:00:52.563915 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:52.565740 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:52.591631 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:52.612709 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:52.614966 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:52.616266 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:52.627660 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:52.629217 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:52.631927 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:52.641530 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:52.641690 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:52.655261 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:52.704401 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:52.706768 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:52.719398 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:52.758206 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:52.786526 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:52.840834 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:52.840931 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:52.845669 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:52.847713 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:52.850423 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:52.853523 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:52.861315 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:52.872282 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:52.876298 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:52.919354 6736 maintenance_manager.cc:419] P f72296bedf9648789ec7fcba5a38bea2: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033385
W20240503 08:00:52.928552 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:52.959211 6735 maintenance_manager.cc:643] P f72296bedf9648789ec7fcba5a38bea2: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.039s user 0.033s sys 0.004s Metrics: {"bytes_written":107970,"cfile_init":1,"dirs.queue_time_us":195,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":3531,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1047,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":301,"thread_start_us":180,"threads_started":2,"wal-append.queue_time_us":148}
I20240503 08:00:52.960251 6736 maintenance_manager.cc:419] P f72296bedf9648789ec7fcba5a38bea2: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 37432 bytes on disk
I20240503 08:00:52.962090 6735 maintenance_manager.cc:643] P f72296bedf9648789ec7fcba5a38bea2: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240503 08:00:53.028928 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:53.029726 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:53.053580 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:53.059924 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:53.060549 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:53.068418 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:53.069455 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:53.083143 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:53.094637 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:53.109378 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:53.131089 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:53.139802 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:53.158753 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:53.178426 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:53.200959 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:53.207713 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:53.221814 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:53.232739 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:53.279306 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:53.283640 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:53.283998 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:53.284974 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:53.288316 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:53.298739 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:53.322801 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:53.328274 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:53.407414 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:53.416970 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:53.499233 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:53.501736 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:53.512884 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:53.514138 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:53.517119 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:53.517840 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:53.531226 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:53.562840 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:53.663760 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:53.677441 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:53.738816 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:53.799036 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:53.801424 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:53.811189 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:53.868121 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:53.903645 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:53.906105 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:53.907337 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:53.907606 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:53.908262 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:53.938203 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:53.939098 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:53.950237 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:53.953729 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:53.979503 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:54.020206 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:54.103708 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:54.106528 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:54.114495 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:54.120572 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:54.126461 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:54.126904 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:54.141108 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:54.144342 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:54.153895 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:54.179975 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:54.225885 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:00:54.236848 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:54.331386 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:54.332767 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:54.333550 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:54.341912 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:54.342722 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:54.348001 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:54.348379 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:54.348552 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:54.376507 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:54.383774 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:54.385381 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:54.498710 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:54.543699 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:54.548106 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:54.554801 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:54.554818 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 08:00:54.556185 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:54.558542 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:54.558718 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:54.558830 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:54.575738 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:54.576157 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:54.579120 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:54.647171 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 08:00:54.665017 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240503 08:00:54.755151 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:54.768941 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:54.769596 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:54.773633 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:00:54.782999 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:54.788161 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:54.793231 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:54.796795 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:54.797093 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:54.802462 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:54.830016 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:54.874269 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:54.880015 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:00:54.912679 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:54.970506 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:54.976275 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:54.980486 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:54.990804 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:54.993788 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:54.995225 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:55.001691 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:55.023028 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:55.044072 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:55.068027 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:00:55.111702 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:55.135408 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:55.154991 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:55.156136 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:00:55.206749 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:55.214969 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:55.235388 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:55.288635 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:55.355412 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:55.369096 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:55.405989 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:55.411743 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:55.415076 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:55.417335 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:55.417403 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:55.426662 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:55.427461 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:55.428949 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:55.451520 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:55.469925 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:55.471393 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:55.472123 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:55.573033 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:55.628080 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:55.629747 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:55.635139 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:55.635195 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:55.641031 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:55.642133 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240503 08:00:55.645630 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:55.654587 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:55.685896 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:55.714216 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:55.758095 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:55.776571 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:55.779983 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:55.790009 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:55.849742 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:55.852454 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:55.853528 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:55.855850 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:00:55.858007 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:55.859337 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:00:55.869290 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:55.869405 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:55.894686 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:55.897531 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:55.970973 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:56.056140 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:56.060698 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:56.065652 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:56.070156 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:56.090082 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:56.090796 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:56.094550 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:56.131184 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:56.174837 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:56.239205 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:56.265264 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:56.288012 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:56.287992 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:00:56.294260 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:56.296073 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:56.297469 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:56.298746 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:56.318257 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:56.319269 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:56.320130 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:56.336185 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:56.416504 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:56.428998 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:56.445005 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:56.469550 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:56.513872 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:56.519058 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:56.520462 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:00:56.522531 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:56.532428 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:56.532837 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:56.536058 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:56.539309 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:56.550901 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:56.552579 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:56.650143 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:56.710565 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:56.769212 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:56.787165 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:56.816064 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:56.847393 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:56.896453 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:00:56.933502 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:56.935647 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:56.937741 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:56.938122 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:56.952577 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:56.969777 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:56.975194 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:00:56.979925 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:57.017484 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:57.100291 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:57.108721 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:00:57.152627 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:00:57.158550 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:57.159477 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:57.165071 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:57.174710 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:57.175474 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:57.179533 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:57.186064 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:57.208976 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:57.255033 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:57.255117 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:57.329109 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:00:57.381755 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:57.383623 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:57.384984 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:57.392346 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:57.403515 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:57.416601 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:57.421579 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:57.524617 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:57.540057 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:57.596827 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:57.596873 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:00:57.609822 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:57.610728 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:57.616736 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:57.623634 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:57.633498 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:57.634270 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:57.634541 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:57.681043 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:57.696643 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:57.750478 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:57.761250 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:57.763859 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:57.814101 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:00:57.828063 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:57.830832 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:57.831184 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:00:57.836973 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:57.840173 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:57.847244 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:57.862115 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:57.864826 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:57.871170 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:57.954224 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:57.984532 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:58.025964 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:00:58.049533 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:58.050665 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:58.056905 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:58.061045 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:58.067215 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:58.071610 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:58.076385 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:58.186275 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:58.200363 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:58.230654 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:58.247325 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:58.259764 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:58.266186 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:58.275846 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:58.278941 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:58.281004 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:58.301826 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:58.328603 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:58.345322 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:58.404855 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:58.465754 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:58.469417 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:58.476557 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:58.476855 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:58.482044 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:58.501951 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:58.514231 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:58.673295 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:58.686170 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:58.689896 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:58.691104 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:58.703045 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:00:58.711009 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:58.722640 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:58.722759 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:58.732316 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:58.732358 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:58.735988 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:58.755460 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:58.770193 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:58.779455 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:00:58.806478 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(5293a26b65484ae5a9e76469e644c5bc): perf score=0.033343
W20240503 08:00:58.811992 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:58.813252 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:58.866441 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(ea625a1834fb4a038a32a90d2fcbf1ac): perf score=0.033357
I20240503 08:00:58.866079 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(5293a26b65484ae5a9e76469e644c5bc) complete. Timing: real 0.059s user 0.035s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":186,"dirs.queue_time_us":15338,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":1489,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":646,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":267,"threads_started":3,"wal-append.queue_time_us":13361}
I20240503 08:00:58.868681 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(5293a26b65484ae5a9e76469e644c5bc): 4838 bytes on disk
I20240503 08:00:58.869410 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(5293a26b65484ae5a9e76469e644c5bc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240503 08:00:58.929684 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:58.933219 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(ea625a1834fb4a038a32a90d2fcbf1ac) complete. Timing: real 0.066s user 0.032s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":15907,"dirs.queue_time_us":206,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":3012,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":621,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":315,"threads_started":3,"wal-append.queue_time_us":186}
I20240503 08:00:58.936076 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(ea625a1834fb4a038a32a90d2fcbf1ac): 4923 bytes on disk
I20240503 08:00:58.937034 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(ea625a1834fb4a038a32a90d2fcbf1ac) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240503 08:00:58.939019 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(660355a4482d4a30bbb71cd0b56527d1): perf score=0.033376
W20240503 08:00:58.972702 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:58.975211 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(5293a26b65484ae5a9e76469e644c5bc): perf score=0.033389
I20240503 08:00:59.029533 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling FlushMRSOp(ea625a1834fb4a038a32a90d2fcbf1ac): perf score=0.033402
W20240503 08:00:59.036310 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:59.038053 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:59.053622 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:59.066419 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:59.069932 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:00:59.070855 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:59.071695 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:59.092136 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:59.094798 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(660355a4482d4a30bbb71cd0b56527d1) complete. Timing: real 0.156s user 0.018s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1760,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":706,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":119,"threads_started":1,"wal-append.queue_time_us":88342}
I20240503 08:00:59.096752 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(660355a4482d4a30bbb71cd0b56527d1): 4581 bytes on disk
I20240503 08:00:59.097744 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(660355a4482d4a30bbb71cd0b56527d1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 08:00:59.098915 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:59.138712 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:59.144990 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(5293a26b65484ae5a9e76469e644c5bc) complete. Timing: real 0.169s user 0.034s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":18649,"dirs.queue_time_us":12532,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1005,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":640,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":551,"threads_started":3,"wal-append.queue_time_us":93135}
I20240503 08:00:59.147630 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(5293a26b65484ae5a9e76469e644c5bc): 4838 bytes on disk
I20240503 08:00:59.148280 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(5293a26b65484ae5a9e76469e644c5bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 08:00:59.150130 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(660355a4482d4a30bbb71cd0b56527d1): perf score=0.033437
W20240503 08:00:59.206257 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:59.219843 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:59.225113 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: FlushMRSOp(ea625a1834fb4a038a32a90d2fcbf1ac) complete. Timing: real 0.195s user 0.030s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":99025,"dirs.queue_time_us":11078,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":878,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":251,"threads_started":3,"wal-append.queue_time_us":14118}
I20240503 08:00:59.227782 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling UndoDeltaBlockGCOp(ea625a1834fb4a038a32a90d2fcbf1ac): 4923 bytes on disk
I20240503 08:00:59.228320 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(660355a4482d4a30bbb71cd0b56527d1) complete. Timing: real 0.078s user 0.019s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1541,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":559,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":101,"threads_started":1,"wal-append.queue_time_us":37187}
I20240503 08:00:59.230477 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(660355a4482d4a30bbb71cd0b56527d1): 4581 bytes on disk
I20240503 08:00:59.232265 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(660355a4482d4a30bbb71cd0b56527d1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240503 08:00:59.233460 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: UndoDeltaBlockGCOp(ea625a1834fb4a038a32a90d2fcbf1ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240503 08:00:59.266413 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:59.274106 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:00:59.312371 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:59.334316 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:59.383147 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:00:59.403556 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:59.518514 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:00:59.544363 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:59.551156 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:59.552623 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:00:59.562552 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:59.567468 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:59.568012 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:59.575947 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:59.577919 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:59.584214 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:59.599057 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240503 08:00:59.659238 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:59.686619 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:00:59.732412 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:59.753181 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:59.755038 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:59.761255 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:59.761396 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 77 similar messages]
I20240503 08:00:59.768548 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:00:59.780474 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:00:59.789050 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:00:59.798276 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:00:59.809091 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:59.820173 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:00:59.852859 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:59.881258 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:00:59.951481 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:00:59.985112 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:00:59.985455 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 102 similar messages]
I20240503 08:00:59.988361 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:00:59.988658 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:00:59.993103 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:00:59.993983 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:00.011531 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:00.035384 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:00.092695 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:00.103232 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:01:00.106132 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(abb55c09be324f37a1a546e73008a80d): perf score=0.033353
I20240503 08:01:00.120174 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(abb55c09be324f37a1a546e73008a80d) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1629,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":464,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":131}
I20240503 08:01:00.121592 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(abb55c09be324f37a1a546e73008a80d): 4838 bytes on disk
I20240503 08:01:00.121958 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(abb55c09be324f37a1a546e73008a80d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240503 08:01:00.130967 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(c27273ae45c449018143cc55e65336a6): perf score=0.033358
I20240503 08:01:00.145044 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(c27273ae45c449018143cc55e65336a6) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1418,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":478,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":133}
I20240503 08:01:00.146783 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(5b794508aa77492398e48d25edfb8aa3): perf score=0.033362
I20240503 08:01:00.161188 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(5b794508aa77492398e48d25edfb8aa3) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1426,"drs_written":1,"lbm_read_time_us":413,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":95,"threads_started":1,"wal-append.queue_time_us":171}
I20240503 08:01:00.162925 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(c27273ae45c449018143cc55e65336a6): 4922 bytes on disk
W20240503 08:01:00.162988 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:00.163412 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(c27273ae45c449018143cc55e65336a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240503 08:01:00.163507 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 08:01:00.165153 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(5b794508aa77492398e48d25edfb8aa3): 4583 bytes on disk
I20240503 08:01:00.165535 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(5b794508aa77492398e48d25edfb8aa3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240503 08:01:00.196678 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:00.210373 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:00.242533 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling FlushMRSOp(abb55c09be324f37a1a546e73008a80d): perf score=0.033388
I20240503 08:01:00.245371 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(c27273ae45c449018143cc55e65336a6): perf score=0.033390
W20240503 08:01:00.252359 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:00.260076 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: FlushMRSOp(abb55c09be324f37a1a546e73008a80d) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":552,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":243,"threads_started":2,"wal-append.queue_time_us":137}
I20240503 08:01:00.262022 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling UndoDeltaBlockGCOp(abb55c09be324f37a1a546e73008a80d): 4838 bytes on disk
I20240503 08:01:00.262462 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: UndoDeltaBlockGCOp(abb55c09be324f37a1a546e73008a80d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 08:01:00.264748 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(c27273ae45c449018143cc55e65336a6) complete. Timing: real 0.019s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1067,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1748,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":608,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":153,"threads_started":2,"wal-append.queue_time_us":152}
I20240503 08:01:00.266387 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(c27273ae45c449018143cc55e65336a6): 4922 bytes on disk
I20240503 08:01:00.266885 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(c27273ae45c449018143cc55e65336a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240503 08:01:00.283460 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:00.323372 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:00.378671 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:00.397533 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:00.408084 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:00.411818 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:00.416970 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:00.419340 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:00.431129 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:00.440207 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:00.448978 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:00.464596 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:00.536389 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:00.573580 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:00.589303 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:00.591166 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:00.621119 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:00.623474 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:00.623772 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:00.625275 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:00.631584 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:00.631558 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:00.634285 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:00.642786 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:00.682053 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:00.695731 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:00.726480 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:00.747772 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:00.749277 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:00.764588 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:00.797986 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:00.811663 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 08:01:00.844764 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:00.846263 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:00.847260 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:00.857297 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:00.866572 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:00.866943 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:00.871593 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:00.873451 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:00.883358 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:00.895818 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:00.954830 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:01.027261 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:01.076885 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:01.081569 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:01.081703 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:01.096985 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:01.097126 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:01.098505 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:01.098592 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:01.115887 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:01.163324 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:01.164407 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:01.171257 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:01.270920 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling FlushMRSOp(31c76213f3ec451084ae7b0551cd7b48): perf score=0.033341
I20240503 08:01:01.277981 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(dd6aa70d9c4c4baea597bbb9788fdb37): perf score=0.033341
W20240503 08:01:01.285895 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:01.292877 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: FlushMRSOp(31c76213f3ec451084ae7b0551cd7b48) complete. Timing: real 0.022s user 0.011s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":524,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1412,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":641,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":176}
I20240503 08:01:01.295106 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling UndoDeltaBlockGCOp(31c76213f3ec451084ae7b0551cd7b48): 4923 bytes on disk
I20240503 08:01:01.295900 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: UndoDeltaBlockGCOp(31c76213f3ec451084ae7b0551cd7b48) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240503 08:01:01.312835 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:01.317564 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:01.322161 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:01.334220 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:01.334985 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(dd6aa70d9c4c4baea597bbb9788fdb37) complete. Timing: real 0.039s user 0.012s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":12047,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":7004,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3552,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":4338,"threads_started":2,"wal-append.queue_time_us":160}
I20240503 08:01:01.335847 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:01.337190 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(3658573037664f03b38fefd31d2fe587): perf score=0.033356
I20240503 08:01:01.355508 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:01.356034 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:01.382051 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(dd6aa70d9c4c4baea597bbb9788fdb37): perf score=0.033371
W20240503 08:01:01.386729 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:01.407755 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(3658573037664f03b38fefd31d2fe587) complete. Timing: real 0.069s user 0.014s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1750,"drs_written":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1026,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":88,"threads_started":1,"wal-append.queue_time_us":9129}
I20240503 08:01:01.408061 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(dd6aa70d9c4c4baea597bbb9788fdb37) complete. Timing: real 0.026s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":580,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":309}
I20240503 08:01:01.409817 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(dd6aa70d9c4c4baea597bbb9788fdb37): 4583 bytes on disk
I20240503 08:01:01.409909 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(3658573037664f03b38fefd31d2fe587): 4838 bytes on disk
I20240503 08:01:01.410691 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(3658573037664f03b38fefd31d2fe587) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 08:01:01.411280 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(dd6aa70d9c4c4baea597bbb9788fdb37) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240503 08:01:01.412328 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(dd6aa70d9c4c4baea597bbb9788fdb37): 4583 bytes on disk
I20240503 08:01:01.412705 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(dd6aa70d9c4c4baea597bbb9788fdb37) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 08:01:01.434607 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(3658573037664f03b38fefd31d2fe587): perf score=0.033385
W20240503 08:01:01.441042 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:01.448254 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:01.449177 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:01.450567 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:01.457158 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(3658573037664f03b38fefd31d2fe587) complete. Timing: real 0.022s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":175,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2035,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":149,"threads_started":2,"wal-append.queue_time_us":147}
I20240503 08:01:01.459252 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(3658573037664f03b38fefd31d2fe587): 4838 bytes on disk
I20240503 08:01:01.461799 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(3658573037664f03b38fefd31d2fe587) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240503 08:01:01.490327 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:01.546913 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:01.565455 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:01.572418 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:01.574836 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:01.576247 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:01.576488 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:01.577575 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:01.613240 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:01.703684 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:01.722380 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:01.783450 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:01.813045 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:01.839936 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:01.947779 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:01.954566 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:01.984493 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:01.984843 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:01.987887 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:01.995051 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:02.005506 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:02.011225 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:02.012676 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:02.015697 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:02.043848 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:02.110057 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:02.131069 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:02.191866 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:02.192191 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:02.199602 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:02.212903 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:02.216073 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:02.229550 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:02.232070 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:02.233418 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:02.262256 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:02.264954 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:02.311033 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:02.327243 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:02.341588 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:02.412974 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:02.413177 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:02.419937 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(7e8e04c50d7147af903d7113df3d9744): perf score=0.033342
I20240503 08:01:02.421761 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(5390e70b0449449aa3af05112bd29797): perf score=0.033340
I20240503 08:01:02.429169 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:02.429153 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:02.439461 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:02.451386 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(7e8e04c50d7147af903d7113df3d9744) complete. Timing: real 0.031s user 0.016s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":601,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":1016}
I20240503 08:01:02.453358 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(7e8e04c50d7147af903d7113df3d9744): 4838 bytes on disk
I20240503 08:01:02.453828 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(7e8e04c50d7147af903d7113df3d9744) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 08:01:02.455063 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:02.472352 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:02.473971 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(7b424a8b7bba44b098731f8ee18e8c9b): perf score=0.033357
W20240503 08:01:02.488816 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:02.504539 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:02.505360 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(7b424a8b7bba44b098731f8ee18e8c9b) complete. Timing: real 0.031s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":897,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":1125,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":542,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":212,"threads_started":2,"wal-append.queue_time_us":186}
I20240503 08:01:02.507470 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(7b424a8b7bba44b098731f8ee18e8c9b): 4923 bytes on disk
I20240503 08:01:02.507978 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(7b424a8b7bba44b098731f8ee18e8c9b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 08:01:02.532589 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(5390e70b0449449aa3af05112bd29797) complete. Timing: real 0.111s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":14212,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1516,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":614,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":192,"threads_started":2,"wal-append.queue_time_us":10671}
I20240503 08:01:02.534591 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(5390e70b0449449aa3af05112bd29797): 4583 bytes on disk
I20240503 08:01:02.535022 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(5390e70b0449449aa3af05112bd29797) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 08:01:02.536787 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(7e8e04c50d7147af903d7113df3d9744): perf score=0.033371
I20240503 08:01:02.554006 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(7e8e04c50d7147af903d7113df3d9744) complete. Timing: real 0.017s user 0.010s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1766,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":564,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":145}
I20240503 08:01:02.555963 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(7e8e04c50d7147af903d7113df3d9744): 4838 bytes on disk
I20240503 08:01:02.556252 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling FlushMRSOp(7b424a8b7bba44b098731f8ee18e8c9b): perf score=0.033376
I20240503 08:01:02.556895 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(7e8e04c50d7147af903d7113df3d9744) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240503 08:01:02.561735 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:02.573135 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: FlushMRSOp(7b424a8b7bba44b098731f8ee18e8c9b) complete. Timing: real 0.017s user 0.012s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":395,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":3314,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":433,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":165,"threads_started":2,"wal-append.queue_time_us":1935}
I20240503 08:01:02.574975 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling UndoDeltaBlockGCOp(7b424a8b7bba44b098731f8ee18e8c9b): 4923 bytes on disk
I20240503 08:01:02.575672 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: UndoDeltaBlockGCOp(7b424a8b7bba44b098731f8ee18e8c9b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240503 08:01:02.669050 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:02.691777 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:02.697379 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:02.698462 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:02.701073 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:02.709863 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:02.710191 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:02.712354 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:02.714711 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:02.716643 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:02.722000 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:02.758087 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:02.788676 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:02.798168 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:02.847687 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:02.887516 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:02.904615 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:02.908478 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:02.913045 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:02.923116 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:02.933533 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:02.934404 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:02.957265 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:02.961364 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:02.968631 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:03.012468 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:03.076545 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:03.093230 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:03.119946 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:03.134482 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:03.139060 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:03.142557 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:03.143456 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:03.153856 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:03.158550 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:03.165489 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:03.184340 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:03.216140 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:03.231472 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:03.326632 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:03.326925 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:03.366822 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:03.382539 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:03.399670 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:03.423769 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:03.430035 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:03.431941 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:03.555891 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:03.557296 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:03.565812 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:03.566944 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:03.569440 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:03.571027 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:03.572006 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:03.594272 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:03.637025 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:03.688040 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:03.695051 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:03.714375 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(0fcc4c2eeaec4b359108ec64a6e20da8): perf score=0.033364
I20240503 08:01:03.737001 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(0fcc4c2eeaec4b359108ec64a6e20da8) complete. Timing: real 0.022s user 0.016s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1551,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":3078,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":525,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":2010}
I20240503 08:01:03.739049 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(aa45918bb38c4583ad59524bda2d48ee): perf score=0.033367
W20240503 08:01:03.744220 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:03.768401 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(aa45918bb38c4583ad59524bda2d48ee): perf score=0.033377
I20240503 08:01:03.774503 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(aa45918bb38c4583ad59524bda2d48ee) complete. Timing: real 0.024s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1462,"drs_written":1,"lbm_read_time_us":151,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1037,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":74,"threads_started":1,"wal-append.queue_time_us":1717}
I20240503 08:01:03.776113 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(0fcc4c2eeaec4b359108ec64a6e20da8): 4923 bytes on disk
I20240503 08:01:03.776510 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(0fcc4c2eeaec4b359108ec64a6e20da8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240503 08:01:03.777876 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(aa45918bb38c4583ad59524bda2d48ee): 4838 bytes on disk
I20240503 08:01:03.778301 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(aa45918bb38c4583ad59524bda2d48ee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240503 08:01:03.784451 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:03.795217 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(aa45918bb38c4583ad59524bda2d48ee) complete. Timing: real 0.027s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":188,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":2193,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":645,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":768,"thread_start_us":168,"threads_started":2,"wal-append.queue_time_us":170}
I20240503 08:01:03.797556 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(aa45918bb38c4583ad59524bda2d48ee): 4838 bytes on disk
I20240503 08:01:03.798375 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(aa45918bb38c4583ad59524bda2d48ee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 08:01:03.805784 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:03.809216 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:03.816524 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(7c2386cd009847d9a1fe7a959f47086c): perf score=0.033394
I20240503 08:01:03.821739 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:03.829798 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:03.832376 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:03.835979 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling FlushMRSOp(7c2386cd009847d9a1fe7a959f47086c): perf score=0.033399
I20240503 08:01:03.837781 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:03.838589 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:03.848227 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:03.857182 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:03.864527 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: FlushMRSOp(7c2386cd009847d9a1fe7a959f47086c) complete. Timing: real 0.028s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3525,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4048,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":276}
I20240503 08:01:03.866561 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling UndoDeltaBlockGCOp(7c2386cd009847d9a1fe7a959f47086c): 4583 bytes on disk
I20240503 08:01:03.867306 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: UndoDeltaBlockGCOp(7c2386cd009847d9a1fe7a959f47086c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240503 08:01:03.879884 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:03.880414 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(7c2386cd009847d9a1fe7a959f47086c) complete. Timing: real 0.063s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":164,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1040,"drs_written":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4,"lbm_write_time_us":552,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":672,"threads_started":2,"wal-append.queue_time_us":5308}
I20240503 08:01:03.882319 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(0fcc4c2eeaec4b359108ec64a6e20da8): perf score=0.033411
I20240503 08:01:03.897372 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(0fcc4c2eeaec4b359108ec64a6e20da8) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":132,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1302,"drs_written":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4,"lbm_write_time_us":492,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":88,"threads_started":1,"wal-append.queue_time_us":287}
I20240503 08:01:03.899119 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(0fcc4c2eeaec4b359108ec64a6e20da8): 4923 bytes on disk
I20240503 08:01:03.899571 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(0fcc4c2eeaec4b359108ec64a6e20da8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240503 08:01:03.901261 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(7c2386cd009847d9a1fe7a959f47086c): 4583 bytes on disk
I20240503 08:01:03.901672 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(7c2386cd009847d9a1fe7a959f47086c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240503 08:01:03.938442 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:04.051057 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:04.051522 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:04.051796 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:04.052695 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:04.057772 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:04.068809 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:04.072032 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:04.072198 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:04.076448 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:04.088321 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:04.126499 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:04.237157 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:04.271854 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:04.275560 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:04.276258 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:04.280927 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:04.280973 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:04.297777 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:04.298096 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:04.316134 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:04.334437 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:04.374770 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:04.377617 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:04.437911 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:04.463881 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:04.495676 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:04.499037 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:04.502733 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:04.504884 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:04.506727 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:04.510394 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:04.518772 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:04.535384 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:04.557829 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:04.603104 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240503 08:01:04.623986 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling FlushMRSOp(f365955974634bc58e8a0da0fc2563d7): perf score=0.033344
I20240503 08:01:04.653095 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: FlushMRSOp(f365955974634bc58e8a0da0fc2563d7) complete. Timing: real 0.029s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":5401,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":191,"threads_started":2,"wal-append.queue_time_us":357}
I20240503 08:01:04.655277 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling FlushMRSOp(921bd35f61ed4465aef45eee3383c9b3): perf score=0.033351
I20240503 08:01:04.658640 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(921bd35f61ed4465aef45eee3383c9b3): perf score=0.033351
W20240503 08:01:04.663273 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:04.689936 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: FlushMRSOp(921bd35f61ed4465aef45eee3383c9b3) complete. Timing: real 0.034s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1443,"drs_written":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4,"lbm_write_time_us":640,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":123,"threads_started":1,"wal-append.queue_time_us":4395}
I20240503 08:01:04.692454 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling UndoDeltaBlockGCOp(f365955974634bc58e8a0da0fc2563d7): 4923 bytes on disk
I20240503 08:01:04.693352 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: UndoDeltaBlockGCOp(f365955974634bc58e8a0da0fc2563d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240503 08:01:04.695158 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling UndoDeltaBlockGCOp(921bd35f61ed4465aef45eee3383c9b3): 4838 bytes on disk
I20240503 08:01:04.701727 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: UndoDeltaBlockGCOp(921bd35f61ed4465aef45eee3383c9b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240503 08:01:04.710933 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:04.712320 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(921bd35f61ed4465aef45eee3383c9b3) complete. Timing: real 0.053s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1463,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1517,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1237,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":217,"threads_started":2,"wal-append.queue_time_us":17704}
I20240503 08:01:04.714308 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(921bd35f61ed4465aef45eee3383c9b3): 4838 bytes on disk
I20240503 08:01:04.714797 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(921bd35f61ed4465aef45eee3383c9b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 08:01:04.741264 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:04.743409 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:04.747066 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:04.747745 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:04.748225 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:04.762574 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240503 08:01:04.775440 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:04.777395 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:04.786729 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240503 08:01:04.786878 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(6e63db30c0454722b6a794a9259a67c0): perf score=0.033388
W20240503 08:01:04.792048 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:04.807668 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(f365955974634bc58e8a0da0fc2563d7): perf score=0.033398
I20240503 08:01:04.812534 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(6e63db30c0454722b6a794a9259a67c0) complete. Timing: real 0.025s user 0.011s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":193,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":4907,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":562,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":162,"threads_started":2,"wal-append.queue_time_us":3294}
I20240503 08:01:04.814551 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(6e63db30c0454722b6a794a9259a67c0): 4583 bytes on disk
I20240503 08:01:04.814961 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(6e63db30c0454722b6a794a9259a67c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240503 08:01:04.817695 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:04.818827 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:04.827093 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(f365955974634bc58e8a0da0fc2563d7) complete. Timing: real 0.019s user 0.014s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":172,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1256,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":511,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":164,"threads_started":2,"wal-append.queue_time_us":137}
I20240503 08:01:04.829057 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(6e63db30c0454722b6a794a9259a67c0): perf score=0.033398
I20240503 08:01:04.843549 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(6e63db30c0454722b6a794a9259a67c0) complete. Timing: real 0.014s user 0.011s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3237,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1267,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":466,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":123,"threads_started":1,"wal-append.queue_time_us":219}
I20240503 08:01:04.845657 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(f365955974634bc58e8a0da0fc2563d7): 4923 bytes on disk
I20240503 08:01:04.846199 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(f365955974634bc58e8a0da0fc2563d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 08:01:04.848096 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(6e63db30c0454722b6a794a9259a67c0): 4583 bytes on disk
I20240503 08:01:04.848639 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(6e63db30c0454722b6a794a9259a67c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 08:01:04.911450 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:04.983137 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:04.993003 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:01:05.011302 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:05.012566 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:05.056161 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:05.092115 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:05.155953 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:01:05.178490 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:05.179276 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 08:01:05.179890 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:05.180150 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:05.184733 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:05.185851 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:05.200718 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:05.202286 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:05.204293 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:05.206554 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:05.243490 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:05.361572 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:05.388674 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:05.402411 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:05.406544 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:05.409739 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:05.422940 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:05.429008 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:05.429008 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:05.433296 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:05.480304 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:05.503858 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:05.520941 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:05.605743 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(eef64b6b6db54db2b47314d672269043): perf score=0.033338
W20240503 08:01:05.614800 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:05.617769 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:05.627439 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(eef64b6b6db54db2b47314d672269043) complete. Timing: real 0.021s user 0.014s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":160,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1475,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":465}
I20240503 08:01:05.629248 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(eef64b6b6db54db2b47314d672269043): 4836 bytes on disk
I20240503 08:01:05.630931 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(eef64b6b6db54db2b47314d672269043) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240503 08:01:05.631839 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:05.632143 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:05.636409 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:05.640182 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:05.640473 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:05.643783 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:05.644268 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:05.658432 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:05.662900 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:05.666493 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:05.672194 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:05.684372 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:05.691061 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:05.710328 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling FlushMRSOp(1ae7895f7d7f47c3ab5fb0b93c1fd8e0): perf score=0.033366
I20240503 08:01:05.723405 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(5365969434f142b8a5757a9a73bc109a): perf score=0.033372
W20240503 08:01:05.725270 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:05.728185 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: FlushMRSOp(1ae7895f7d7f47c3ab5fb0b93c1fd8e0) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":568,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":170,"threads_started":2,"wal-append.queue_time_us":172}
I20240503 08:01:05.730074 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling UndoDeltaBlockGCOp(1ae7895f7d7f47c3ab5fb0b93c1fd8e0): 4583 bytes on disk
I20240503 08:01:05.730610 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: UndoDeltaBlockGCOp(1ae7895f7d7f47c3ab5fb0b93c1fd8e0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 08:01:05.732712 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling FlushMRSOp(eef64b6b6db54db2b47314d672269043): perf score=0.033370
W20240503 08:01:05.745674 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:05.755045 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(5365969434f142b8a5757a9a73bc109a) complete. Timing: real 0.031s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1573,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":138}
I20240503 08:01:05.756898 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: FlushMRSOp(eef64b6b6db54db2b47314d672269043) complete. Timing: real 0.024s user 0.011s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1661,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":628,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":69,"threads_started":1,"wal-append.queue_time_us":5454}
I20240503 08:01:05.757241 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(5365969434f142b8a5757a9a73bc109a): 4921 bytes on disk
I20240503 08:01:05.757664 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(5365969434f142b8a5757a9a73bc109a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 08:01:05.758589 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling UndoDeltaBlockGCOp(eef64b6b6db54db2b47314d672269043): 4836 bytes on disk
I20240503 08:01:05.759100 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: UndoDeltaBlockGCOp(eef64b6b6db54db2b47314d672269043) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 08:01:05.823982 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(1ae7895f7d7f47c3ab5fb0b93c1fd8e0): perf score=0.033395
W20240503 08:01:05.830516 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:05.857456 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(1ae7895f7d7f47c3ab5fb0b93c1fd8e0) complete. Timing: real 0.033s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1933,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":723,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":254,"threads_started":2,"wal-append.queue_time_us":8089}
I20240503 08:01:05.859397 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(1ae7895f7d7f47c3ab5fb0b93c1fd8e0): 4583 bytes on disk
I20240503 08:01:05.861304 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(1ae7895f7d7f47c3ab5fb0b93c1fd8e0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240503 08:01:05.880204 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:05.881682 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:05.882277 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:05.888917 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:05.903149 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:05.905050 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:05.909456 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:05.921622 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:05.931344 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:05.932540 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:05.979735 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:05.980300 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:05.986297 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:01:06.061535 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:06.101935 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:06.103130 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:06.107309 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:06.116428 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:06.118525 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:06.123015 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:06.125340 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:06.133116 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:06.327932 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:06.331926 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:06.332442 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:06.335801 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:06.337525 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:06.340858 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:06.352229 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:06.353878 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:06.407636 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:06.407780 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:06.427546 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:06.462716 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:06.471913 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:06.476779 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:06.488068 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:06.498381 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:06.560132 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:06.689018 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:06.720805 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:06.722858 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:06.732975 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:06.743664 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:06.748183 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:06.749786 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:06.756881 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:06.767186 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(16c06ab5a3e54ed2910afa50069810cc): perf score=0.033343
I20240503 08:01:06.767454 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:06.771586 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling FlushMRSOp(cb2190946fb04a6392097f9588ffda6b): perf score=0.033351
I20240503 08:01:06.771827 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:06.791971 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: FlushMRSOp(cb2190946fb04a6392097f9588ffda6b) complete. Timing: real 0.020s user 0.015s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1076,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":153}
I20240503 08:01:06.793826 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling UndoDeltaBlockGCOp(cb2190946fb04a6392097f9588ffda6b): 4923 bytes on disk
W20240503 08:01:06.794224 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:06.795892 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: UndoDeltaBlockGCOp(cb2190946fb04a6392097f9588ffda6b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 08:01:06.799711 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:06.809334 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:06.832617 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(16c06ab5a3e54ed2910afa50069810cc) complete. Timing: real 0.064s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4017,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1348,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":156,"threads_started":2,"wal-append.queue_time_us":17379}
I20240503 08:01:06.834535 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(16c06ab5a3e54ed2910afa50069810cc): 4838 bytes on disk
I20240503 08:01:06.834977 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(16c06ab5a3e54ed2910afa50069810cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 08:01:06.871450 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(795f69249afe444ba0828875e1839b8c): perf score=0.033372
W20240503 08:01:06.877321 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:06.889972 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(795f69249afe444ba0828875e1839b8c) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":611,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":131,"threads_started":2,"wal-append.queue_time_us":748}
I20240503 08:01:06.891642 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(cb2190946fb04a6392097f9588ffda6b): perf score=0.033384
I20240503 08:01:06.891806 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(795f69249afe444ba0828875e1839b8c): 4583 bytes on disk
I20240503 08:01:06.892262 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(795f69249afe444ba0828875e1839b8c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 08:01:06.910923 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(cb2190946fb04a6392097f9588ffda6b) complete. Timing: real 0.019s user 0.013s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":184,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1232,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":537,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":153}
I20240503 08:01:06.912734 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(795f69249afe444ba0828875e1839b8c): perf score=0.033389
I20240503 08:01:06.931099 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(795f69249afe444ba0828875e1839b8c) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":207,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1884,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":614,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":163}
I20240503 08:01:06.933157 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(cb2190946fb04a6392097f9588ffda6b): 4923 bytes on disk
I20240503 08:01:06.934607 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(cb2190946fb04a6392097f9588ffda6b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 08:01:06.936547 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(795f69249afe444ba0828875e1839b8c): 4583 bytes on disk
I20240503 08:01:06.937319 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(795f69249afe444ba0828875e1839b8c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240503 08:01:06.947989 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:06.994933 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:07.001441 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:07.009692 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:07.014149 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:07.021996 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:07.030921 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:07.032464 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:07.050004 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:07.069042 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:07.075170 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:07.090220 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:07.166849 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:07.223270 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:07.229174 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:07.235471 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:07.256484 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:07.260376 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:07.262450 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:07.288861 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:07.335216 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:07.438449 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:07.439806 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:07.450155 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:07.451021 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:07.453683 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:07.462706 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:07.470386 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:07.479588 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:07.480494 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:07.480723 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:07.546851 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:07.547307 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:07.567314 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:07.585494 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:07.652396 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:07.656875 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:07.663092 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:07.672271 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:07.675513 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:07.696240 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:07.713734 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:07.741271 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:07.789248 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:07.791837 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:07.812325 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:07.829421 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:07.885399 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:07.890846 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:07.894521 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:07.898142 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:07.898615 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:07.906570 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:07.927798 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:07.968528 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:07.989770 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:08.018038 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:08.055858 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling FlushMRSOp(7d3eb284c0fc48b4912168c434844284): perf score=0.033357
I20240503 08:01:08.069054 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: FlushMRSOp(7d3eb284c0fc48b4912168c434844284) complete. Timing: real 0.013s user 0.010s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":197,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1288,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":466,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":138}
I20240503 08:01:08.070817 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling UndoDeltaBlockGCOp(7d3eb284c0fc48b4912168c434844284): 4838 bytes on disk
I20240503 08:01:08.071223 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: UndoDeltaBlockGCOp(7d3eb284c0fc48b4912168c434844284) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240503 08:01:08.079165 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:08.081147 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:08.095957 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(ba1de17d15b74c4497e649aadc8569d2): perf score=0.033372
I20240503 08:01:08.108834 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(ba1de17d15b74c4497e649aadc8569d2) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1524,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":436,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":170,"threads_started":2,"wal-append.queue_time_us":160}
I20240503 08:01:08.110407 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(ba1de17d15b74c4497e649aadc8569d2): 4923 bytes on disk
I20240503 08:01:08.110781 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(ba1de17d15b74c4497e649aadc8569d2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240503 08:01:08.112381 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(64d33ff4164b49a7948aa5afee40bf3f): perf score=0.033372
I20240503 08:01:08.125975 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(64d33ff4164b49a7948aa5afee40bf3f) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1207,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":498,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":1408,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":470}
I20240503 08:01:08.127579 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(64d33ff4164b49a7948aa5afee40bf3f): 4581 bytes on disk
I20240503 08:01:08.128015 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(64d33ff4164b49a7948aa5afee40bf3f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240503 08:01:08.150099 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:08.153198 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(ba1de17d15b74c4497e649aadc8569d2): perf score=0.033386
I20240503 08:01:08.166457 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(ba1de17d15b74c4497e649aadc8569d2) complete. Timing: real 0.013s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":180,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1462,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":448,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":157,"threads_started":2,"wal-append.queue_time_us":183}
I20240503 08:01:08.168337 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(ba1de17d15b74c4497e649aadc8569d2): 4923 bytes on disk
I20240503 08:01:08.168967 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(ba1de17d15b74c4497e649aadc8569d2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240503 08:01:08.183969 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:08.198724 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(7d3eb284c0fc48b4912168c434844284): perf score=0.033396
I20240503 08:01:08.214306 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(7d3eb284c0fc48b4912168c434844284) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":174,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1293,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":484,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":1280,"thread_start_us":162,"threads_started":2,"wal-append.queue_time_us":164}
I20240503 08:01:08.215978 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(7d3eb284c0fc48b4912168c434844284): 4838 bytes on disk
I20240503 08:01:08.216454 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(7d3eb284c0fc48b4912168c434844284) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240503 08:01:08.233321 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:08.272588 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:08.301249 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:08.303269 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:08.328881 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:08.328982 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:08.329460 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:08.331269 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:08.332757 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:08.343237 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:08.348598 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:08.367261 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:08.367674 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:08.379942 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:08.426674 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:08.465360 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:08.489413 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:08.534713 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:08.546483 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:08.546600 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:08.550155 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:08.552044 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:08.558478 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:08.560714 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:08.562705 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:08.576238 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:08.595625 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:08.613401 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:08.694736 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:08.707087 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:08.764556 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:08.766620 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:08.776217 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:08.778509 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:08.779507 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:08.785001 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:08.789412 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:08.792703 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:08.824174 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:08.831099 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:08.834815 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:08.946660 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:08.961436 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:08.981173 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:08.990602 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:08.998930 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:08.999081 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:09.010522 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:09.021670 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:09.025739 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:09.032505 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:09.037660 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:09.086812 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:09.099056 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:09.112033 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:09.137028 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(82fc9f4263244c6fa2fb39543230eb87): perf score=0.033344
I20240503 08:01:09.161921 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(82fc9f4263244c6fa2fb39543230eb87) complete. Timing: real 0.025s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1073,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1489,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":632,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":212,"threads_started":2,"wal-append.queue_time_us":642}
I20240503 08:01:09.165006 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(82fc9f4263244c6fa2fb39543230eb87): 4838 bytes on disk
I20240503 08:01:09.165580 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(82fc9f4263244c6fa2fb39543230eb87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 08:01:09.188338 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(721c34a8b6a348e5890d462ba7da938c): perf score=0.033359
W20240503 08:01:09.190105 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:09.206665 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(721c34a8b6a348e5890d462ba7da938c) complete. Timing: real 0.018s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1413,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":153,"threads_started":2,"wal-append.queue_time_us":141}
I20240503 08:01:09.209638 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling FlushMRSOp(7bbaf13cbb474734a961664ae8ebe779): perf score=0.033359
W20240503 08:01:09.211839 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:09.225375 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(7bbaf13cbb474734a961664ae8ebe779): perf score=0.033367
I20240503 08:01:09.241381 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: FlushMRSOp(7bbaf13cbb474734a961664ae8ebe779) complete. Timing: real 0.031s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2366,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1541,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":933,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":74,"threads_started":1,"wal-append.queue_time_us":1060}
I20240503 08:01:09.243652 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(7bbaf13cbb474734a961664ae8ebe779): 4923 bytes on disk
I20240503 08:01:09.244863 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(7bbaf13cbb474734a961664ae8ebe779) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 08:01:09.246656 7329 maintenance_manager.cc:419] P 5227b26243484bbfb177a995ceb50afc: Scheduling UndoDeltaBlockGCOp(721c34a8b6a348e5890d462ba7da938c): 4583 bytes on disk
I20240503 08:01:09.247382 7219 maintenance_manager.cc:643] P 5227b26243484bbfb177a995ceb50afc: UndoDeltaBlockGCOp(721c34a8b6a348e5890d462ba7da938c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 08:01:09.249058 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:09.252187 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:09.256381 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:09.257128 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:09.260553 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:09.265022 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(7bbaf13cbb474734a961664ae8ebe779) complete. Timing: real 0.039s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":164,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":1808,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":563,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":2304,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":124}
I20240503 08:01:09.266523 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling FlushMRSOp(721c34a8b6a348e5890d462ba7da938c): perf score=0.033378
I20240503 08:01:09.288204 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: FlushMRSOp(721c34a8b6a348e5890d462ba7da938c) complete. Timing: real 0.021s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":5290,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":643,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":2148,"threads_started":1,"wal-append.queue_time_us":3717}
I20240503 08:01:09.290082 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(7bbaf13cbb474734a961664ae8ebe779): 4923 bytes on disk
I20240503 08:01:09.290593 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(7bbaf13cbb474734a961664ae8ebe779) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 08:01:09.291390 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:09.292217 6936 maintenance_manager.cc:419] P c4218904dd7e42d2be95b901a6e5db92: Scheduling UndoDeltaBlockGCOp(721c34a8b6a348e5890d462ba7da938c): 4583 bytes on disk
I20240503 08:01:09.292619 6826 maintenance_manager.cc:643] P c4218904dd7e42d2be95b901a6e5db92: UndoDeltaBlockGCOp(721c34a8b6a348e5890d462ba7da938c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240503 08:01:09.293728 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:09.311987 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:09.331346 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling FlushMRSOp(82fc9f4263244c6fa2fb39543230eb87): perf score=0.033396
W20240503 08:01:09.340029 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:09.351837 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: FlushMRSOp(82fc9f4263244c6fa2fb39543230eb87) complete. Timing: real 0.020s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":179,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1242,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":483,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":176,"threads_started":2,"wal-append.queue_time_us":169}
I20240503 08:01:09.353675 7067 maintenance_manager.cc:419] P 9c085a3241f14cb7ba1332b508f4b57d: Scheduling UndoDeltaBlockGCOp(82fc9f4263244c6fa2fb39543230eb87): 4838 bytes on disk
I20240503 08:01:09.354094 6957 maintenance_manager.cc:643] P 9c085a3241f14cb7ba1332b508f4b57d: UndoDeltaBlockGCOp(82fc9f4263244c6fa2fb39543230eb87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240503 08:01:09.414678 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:09.515672 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:09.518462 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:09.521571 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:09.521773 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:09.524677 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:09.525318 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:09.532265 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:09.554608 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:09.623747 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:09.627403 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:01:09.641778 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:09.643473 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:09.674517 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:09.786536 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:09.792490 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 08:01:09.799489 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:09.800109 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 08:01:09.808173 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:09.925222 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:09.930953 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:09.931300 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:09.934469 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:09.936409 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:09.938133 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:09.956907 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:09.969667 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:10.036161 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.043583 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 08:01:10.064147 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.069211 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:10.133138 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:10.140467 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:10.141093 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:10.142217 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:10.151526 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:10.159540 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:10.165732 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:10.170910 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:10.172421 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:10.192729 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.210167 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240503 08:01:10.210920 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.234699 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.267824 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:10.334687 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:01:10.360391 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:10.360680 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:10.366412 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:10.369168 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:10.371618 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:10.372882 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:10.397964 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:10.398231 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:10.415030 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.490104 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.526065 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:10.544355 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:10.585224 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:10.585510 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:10.592171 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:10.594378 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:10.597106 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:10.598403 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:10.623526 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:10.652523 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.725610 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.731540 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.772171 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:10.801936 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.802235 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:10.806082 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:10.806375 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:10.806740 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:10.807462 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:10.811239 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:10.826838 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:10.830883 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:10.841055 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:10.849453 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:10.872651 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:10.917398 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.973421 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:10.977392 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:11.028990 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:01:11.031996 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:11.032790 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:11.039716 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:11.040701 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:11.043605 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:11.043977 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:11.056041 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:11.061601 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:11.078528 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:11.085979 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:11.115759 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:11.135192 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:11.147748 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:11.236423 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:11.294409 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:11.326761 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:11.370515 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:11.405422 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:11.450371 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:11.466192 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:11.468240 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:11.476804 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:11.482023 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:11.486157 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:11.492838 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:11.503473 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:11.508538 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:11.510572 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:11.517153 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:11.582552 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:11.635429 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:11.679490 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:11.688303 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:11.689702 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:11.693261 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:11.695863 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:11.700054 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:11.710121 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:11.711421 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:11.712144 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:11.759721 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:11.776250 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:11.898608 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:11.901422 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:11.901639 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:11.906631 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:11.914678 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:11.915006 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:11.919695 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:11.926779 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:11.940690 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:11.987808 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:12.039095 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:12.111407 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:12.123121 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:12.127709 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:12.133015 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:12.138166 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:12.142518 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:12.145288 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:12.146767 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:12.166080 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:12.211645 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:12.253168 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:12.265447 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:12.310972 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:12.320436 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling FlushMRSOp(dd6aa70d9c4c4baea597bbb9788fdb37): perf score=0.033334
I20240503 08:01:12.347869 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:12.359542 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:12.366326 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: FlushMRSOp(dd6aa70d9c4c4baea597bbb9788fdb37) complete. Timing: real 0.045s user 0.025s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":12479,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":660,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":280,"threads_started":3,"wal-append.queue_time_us":155}
I20240503 08:01:12.368858 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling FlushMRSOp(64d33ff4164b49a7948aa5afee40bf3f): perf score=0.033335
I20240503 08:01:12.372949 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:12.390497 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:12.390537 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:12.410317 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:12.416946 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:12.417654 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:12.426215 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: FlushMRSOp(64d33ff4164b49a7948aa5afee40bf3f) complete. Timing: real 0.055s user 0.015s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1291,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":629,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":109,"threads_started":1,"wal-append.queue_time_us":25848}
I20240503 08:01:12.428472 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling UndoDeltaBlockGCOp(dd6aa70d9c4c4baea597bbb9788fdb37): 4583 bytes on disk
I20240503 08:01:12.429250 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: UndoDeltaBlockGCOp(dd6aa70d9c4c4baea597bbb9788fdb37) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 08:01:12.440153 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling UndoDeltaBlockGCOp(64d33ff4164b49a7948aa5afee40bf3f): 4581 bytes on disk
I20240503 08:01:12.440702 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: UndoDeltaBlockGCOp(64d33ff4164b49a7948aa5afee40bf3f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240503 08:01:12.445020 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:12.456991 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling FlushMRSOp(921bd35f61ed4465aef45eee3383c9b3): perf score=0.033367
I20240503 08:01:12.493387 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: FlushMRSOp(921bd35f61ed4465aef45eee3383c9b3) complete. Timing: real 0.036s user 0.027s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":8455,"dirs.queue_time_us":5174,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":858,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":537,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":232,"threads_started":3,"wal-append.queue_time_us":128}
W20240503 08:01:12.495180 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:12.495643 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling UndoDeltaBlockGCOp(921bd35f61ed4465aef45eee3383c9b3): 4838 bytes on disk
I20240503 08:01:12.496213 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: UndoDeltaBlockGCOp(921bd35f61ed4465aef45eee3383c9b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240503 08:01:12.527180 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:12.564560 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:12.585450 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:12.687269 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:12.710943 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:12.726749 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:12.755669 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling FlushMRSOp(eef64b6b6db54db2b47314d672269043): perf score=0.033392
I20240503 08:01:12.760980 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:12.765017 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:12.769773 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:12.777743 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:12.787918 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:12.788858 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:12.806090 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:12.814538 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:12.828485 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: FlushMRSOp(eef64b6b6db54db2b47314d672269043) complete. Timing: real 0.070s user 0.014s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":34240,"compiler_manager_pool.run_cpu_time_us":15,"compiler_manager_pool.run_wall_time_us":15,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1847,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":526,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":117,"threads_started":1,"wal-append.queue_time_us":2039}
I20240503 08:01:12.830478 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling UndoDeltaBlockGCOp(eef64b6b6db54db2b47314d672269043): 4836 bytes on disk
I20240503 08:01:12.830950 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: UndoDeltaBlockGCOp(eef64b6b6db54db2b47314d672269043) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 08:01:12.831485 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:12.937516 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:12.980440 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:12.993563 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:13.012176 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:13.065663 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:13.085285 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:13.098953 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:13.172133 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:13.181519 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:13.182490 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:13.189002 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:13.192878 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:13.206800 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:13.206976 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:13.208042 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling FlushMRSOp(82fc9f4263244c6fa2fb39543230eb87): perf score=0.033341
W20240503 08:01:13.233073 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:13.236688 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:13.254607 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling FlushMRSOp(7b424a8b7bba44b098731f8ee18e8c9b): perf score=0.033397
I20240503 08:01:13.256568 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: FlushMRSOp(82fc9f4263244c6fa2fb39543230eb87) complete. Timing: real 0.048s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":734,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":3749}
I20240503 08:01:13.258127 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling UndoDeltaBlockGCOp(82fc9f4263244c6fa2fb39543230eb87): 4838 bytes on disk
I20240503 08:01:13.258631 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: UndoDeltaBlockGCOp(82fc9f4263244c6fa2fb39543230eb87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240503 08:01:13.261175 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:13.273352 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:13.291471 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:13.291877 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:13.321015 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: FlushMRSOp(7b424a8b7bba44b098731f8ee18e8c9b) complete. Timing: real 0.066s user 0.027s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":26719,"dirs.queue_time_us":160,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":822,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":603,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":244,"threads_started":3,"wal-append.queue_time_us":1188}
I20240503 08:01:13.323843 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling UndoDeltaBlockGCOp(7b424a8b7bba44b098731f8ee18e8c9b): 4923 bytes on disk
I20240503 08:01:13.325832 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: UndoDeltaBlockGCOp(7b424a8b7bba44b098731f8ee18e8c9b) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 08:01:13.336174 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling FlushMRSOp(7e8e04c50d7147af903d7113df3d9744): perf score=0.033351
I20240503 08:01:13.353628 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: FlushMRSOp(7e8e04c50d7147af903d7113df3d9744) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3696,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":651,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":312}
I20240503 08:01:13.355847 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling UndoDeltaBlockGCOp(7e8e04c50d7147af903d7113df3d9744): 4838 bytes on disk
I20240503 08:01:13.359535 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: UndoDeltaBlockGCOp(7e8e04c50d7147af903d7113df3d9744) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240503 08:01:13.396565 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:13.413724 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:13.440156 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:13.456394 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:13.494728 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:13.499135 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:13.499248 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:13.519729 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:13.523978 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:13.527882 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:13.534036 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:13.573454 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:13.579007 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling FlushMRSOp(31c76213f3ec451084ae7b0551cd7b48): perf score=0.033399
W20240503 08:01:13.582827 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:13.601838 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: FlushMRSOp(31c76213f3ec451084ae7b0551cd7b48) complete. Timing: real 0.023s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1352,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":458,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":108,"threads_started":1,"wal-append.queue_time_us":213}
I20240503 08:01:13.605095 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling UndoDeltaBlockGCOp(31c76213f3ec451084ae7b0551cd7b48): 4923 bytes on disk
I20240503 08:01:13.605711 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: UndoDeltaBlockGCOp(31c76213f3ec451084ae7b0551cd7b48) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240503 08:01:13.609112 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:13.611864 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:13.712321 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:13.739732 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:13.746546 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:13.750831 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:13.755771 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:13.771734 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:13.774057 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:13.780123 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:13.786967 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:13.799890 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:13.808107 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:13.872997 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:13.907999 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:13.964771 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:13.968710 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:13.970775 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:13.982072 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:13.989187 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:13.994529 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:13.997329 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:14.004648 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:14.010104 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:14.015003 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:14.019505 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling FlushMRSOp(5365969434f142b8a5757a9a73bc109a): perf score=0.033351
W20240503 08:01:14.027896 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:14.048552 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:14.058887 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: FlushMRSOp(5365969434f142b8a5757a9a73bc109a) complete. Timing: real 0.039s user 0.016s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1345,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":724,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":144}
I20240503 08:01:14.060746 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling UndoDeltaBlockGCOp(5365969434f142b8a5757a9a73bc109a): 4921 bytes on disk
I20240503 08:01:14.061195 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: UndoDeltaBlockGCOp(5365969434f142b8a5757a9a73bc109a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 08:01:14.061686 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:14.121825 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling FlushMRSOp(16c06ab5a3e54ed2910afa50069810cc): perf score=0.033361
W20240503 08:01:14.127274 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:14.139679 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: FlushMRSOp(16c06ab5a3e54ed2910afa50069810cc) complete. Timing: real 0.018s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":214,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1351,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":656,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":182,"threads_started":2,"wal-append.queue_time_us":156}
I20240503 08:01:14.141458 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling UndoDeltaBlockGCOp(16c06ab5a3e54ed2910afa50069810cc): 4838 bytes on disk
I20240503 08:01:14.145097 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: UndoDeltaBlockGCOp(16c06ab5a3e54ed2910afa50069810cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240503 08:01:14.202872 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:14.206830 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:14.211194 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:14.211644 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:14.214962 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:14.230551 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:14.238559 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:14.239271 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:14.251870 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:14.252807 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:14.377131 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:14.388692 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:14.432032 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:14.439116 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:14.453632 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:14.453971 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:14.458544 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:14.463961 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:14.468148 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:14.493746 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:14.558017 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:14.566006 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:14.566510 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:14.604959 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:14.678805 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:14.703717 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240503 08:01:14.746826 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:14.824234 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:14.846151 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 86 similar messages]
I20240503 08:01:14.885926 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:14.886265 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:14.889372 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:14.895833 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:14.896608 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:14.900158 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:14.901674 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:14.988456 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:14.990078 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 08:01:15.060194 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240503 08:01:15.102003 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:15.103549 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:15.110750 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:15.112692 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:15.128687 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:15.130280 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:15.132251 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.137228 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:15.142508 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:15.150323 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:15.158298 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.202706 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.202875 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.320725 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:01:15.323832 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:15.338168 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:15.347122 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:15.354558 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:15.358757 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:15.359330 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:15.360054 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:15.369450 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:15.415604 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:15.420697 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:15.436213 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.460834 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:01:15.489300 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.489300 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.491340 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.515872 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:15.533738 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.539848 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:15.573654 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:15.582607 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:15.586408 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:15.586510 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:15.592183 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:15.594604 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:15.598660 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:15.599439 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.606791 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:15.662209 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.711740 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:15.725666 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.734918 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.765774 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:15.809173 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:15.809912 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:15.812960 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:15.822468 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:15.833882 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:15.836838 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:15.838440 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:15.838572 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:15.890556 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.923115 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:15.928189 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:16.037379 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:16.037716 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:16.044013 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:16.050555 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:16.051290 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:16.054021 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:16.057292 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 08:01:16.059528 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:16.059967 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:16.071113 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:16.080293 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:16.117280 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:16.185204 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:16.189254 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:16.210669 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:16.323081 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:16.358428 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:16.360520 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:16.442835 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:16.463804 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:16.479905 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:16.495682 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:16.505986 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:16.512308 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:16.517084 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:16.514283 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:16.522001 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:16.530556 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:16.534492 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:16.547242 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:16.587335 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:16.618849 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:16.638599 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:16.645936 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:16.681434 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:16.694752 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:16.717185 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:16.717249 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:16.717484 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:16.730948 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:16.733906 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:16.737883 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:16.766944 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:16.848352 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:16.924962 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:16.928951 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:16.937079 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:16.938685 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:16.940812 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:16.944460 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:16.953233 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:16.958564 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:16.995275 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:17.009254 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:17.103545 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.106758 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:17.147580 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:17.147950 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:17.149721 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:17.157253 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:17.164728 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:17.173931 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:17.186064 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:17.194447 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:17.214516 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.339615 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:17.363541 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:17.366149 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:17.368065 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:17.375566 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:17.387451 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:17.388484 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:17.394110 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:17.438970 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.443029 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:17.483848 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.561187 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:17.586977 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:17.587694 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:17.592968 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:17.600150 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:17.607137 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:17.626694 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:17.630810 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:17.669709 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.777221 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.826443 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.843488 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.851325 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.884855 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.886407 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:17.919111 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.926255 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:17.927718 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.951490 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:17.981817 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:18.008424 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:18.047652 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:18.053799 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:18.053799 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:18.060865 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:18.063086 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:18.070648 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:18.071966 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:18.099886 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:18.121397 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:18.207698 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:18.212044 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:18.220068 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:18.268219 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:18.281960 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:18.282238 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:18.283605 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:18.284075 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:18.285043 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:18.288892 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:18.288926 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:18.289325 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:18.349968 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:18.415855 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:18.438843 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:18.447527 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:18.487782 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:18.503504 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:18.510249 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:18.511807 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:18.513989 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:18.524911 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:18.536027 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:18.537133 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:18.547837 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:18.548597 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:18.549165 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:18.618657 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:18.627774 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:18.652421 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:18.692848 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:18.724370 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:18.727392 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:18.733777 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:18.736996 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:18.737768 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:18.767135 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:18.767567 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:18.774400 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:18.828256 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:18.889058 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:18.900807 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:18.949272 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:18.951850 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:18.953379 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:18.963444 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:18.966006 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:18.979434 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:18.990006 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:18.996347 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:19.020514 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:19.090243 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:19.134764 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:19.155484 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:19.160032 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:19.184958 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:19.192277 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:19.193004 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:19.200366 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:19.200629 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:19.200987 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:19.201253 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:19.244585 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:19.254518 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:19.254638 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:19.391383 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:19.422070 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:19.449172 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:19.469378 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:19.481297 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:19.572928 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:19.607542 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:19.611344 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:19.618448 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:19.620167 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:19.620182 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:19.622718 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:19.635434 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:19.667198 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:19.703174 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:19.726516 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 08:01:19.769793 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:19.782723 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:19.814253 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:19.841964 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:19.843154 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:19.846132 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:19.861029 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:19.864413 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:19.864672 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:19.869279 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240503 08:01:19.873044 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:19.885885 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:19.936738 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling FlushMRSOp(7c2386cd009847d9a1fe7a959f47086c): perf score=0.033349
I20240503 08:01:19.953881 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: FlushMRSOp(7c2386cd009847d9a1fe7a959f47086c) complete. Timing: real 0.017s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":337,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1490,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":467,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":203,"threads_started":2,"wal-append.queue_time_us":197}
I20240503 08:01:19.955819 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling UndoDeltaBlockGCOp(7c2386cd009847d9a1fe7a959f47086c): 4583 bytes on disk
I20240503 08:01:19.956461 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: UndoDeltaBlockGCOp(7c2386cd009847d9a1fe7a959f47086c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 08:01:20.071619 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:20.075857 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:20.082088 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:20.082875 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:20.093135 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:20.098965 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:20.109027 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling FlushMRSOp(c27273ae45c449018143cc55e65336a6): perf score=0.033396
I20240503 08:01:20.113564 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:20.128491 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:20.128754 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 08:01:20.148612 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240503 08:01:20.149274 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:20.154844 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: FlushMRSOp(c27273ae45c449018143cc55e65336a6) complete. Timing: real 0.046s user 0.017s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2740,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":3628,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":702,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":171,"threads_started":2,"wal-append.queue_time_us":169}
I20240503 08:01:20.156857 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling FlushMRSOp(aa45918bb38c4583ad59524bda2d48ee): perf score=0.033362
I20240503 08:01:20.157985 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling FlushMRSOp(ea625a1834fb4a038a32a90d2fcbf1ac): perf score=0.033365
I20240503 08:01:20.179648 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: FlushMRSOp(ea625a1834fb4a038a32a90d2fcbf1ac) complete. Timing: real 0.021s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":195,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1284,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":455,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":915}
I20240503 08:01:20.187361 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling UndoDeltaBlockGCOp(ea625a1834fb4a038a32a90d2fcbf1ac): 4923 bytes on disk
I20240503 08:01:20.187798 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: UndoDeltaBlockGCOp(ea625a1834fb4a038a32a90d2fcbf1ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 08:01:20.204246 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: FlushMRSOp(aa45918bb38c4583ad59524bda2d48ee) complete. Timing: real 0.047s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1495,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":481,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":100,"threads_started":1,"wal-append.queue_time_us":1590}
I20240503 08:01:20.206781 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling UndoDeltaBlockGCOp(c27273ae45c449018143cc55e65336a6): 4922 bytes on disk
I20240503 08:01:20.207304 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: UndoDeltaBlockGCOp(c27273ae45c449018143cc55e65336a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 08:01:20.209237 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling UndoDeltaBlockGCOp(aa45918bb38c4583ad59524bda2d48ee): 4838 bytes on disk
I20240503 08:01:20.209658 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: UndoDeltaBlockGCOp(aa45918bb38c4583ad59524bda2d48ee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240503 08:01:20.209785 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling FlushMRSOp(5b794508aa77492398e48d25edfb8aa3): perf score=0.033353
W20240503 08:01:20.217482 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:20.227289 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: FlushMRSOp(5b794508aa77492398e48d25edfb8aa3) complete. Timing: real 0.017s user 0.014s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1440,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":88,"threads_started":1,"wal-append.queue_time_us":163}
I20240503 08:01:20.229151 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling UndoDeltaBlockGCOp(5b794508aa77492398e48d25edfb8aa3): 4583 bytes on disk
I20240503 08:01:20.229617 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: UndoDeltaBlockGCOp(5b794508aa77492398e48d25edfb8aa3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 08:01:20.231369 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling FlushMRSOp(3658573037664f03b38fefd31d2fe587): perf score=0.033335
W20240503 08:01:20.247332 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:20.259512 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: FlushMRSOp(3658573037664f03b38fefd31d2fe587) complete. Timing: real 0.027s user 0.016s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1556,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":615,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":131,"threads_started":1,"wal-append.queue_time_us":3104}
I20240503 08:01:20.261664 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling UndoDeltaBlockGCOp(3658573037664f03b38fefd31d2fe587): 4838 bytes on disk
I20240503 08:01:20.263856 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: UndoDeltaBlockGCOp(3658573037664f03b38fefd31d2fe587) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240503 08:01:20.293011 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:20.294538 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:20.318742 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:20.330507 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:20.334913 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:20.344893 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:20.350988 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:20.356554 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:20.358529 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:20.366179 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:20.368683 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:20.383818 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:20.394464 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:20.397693 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:20.418272 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:20.482012 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 08:01:20.482760 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:20.498225 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:01:20.520016 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:20.522298 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:20.529876 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:20.559861 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:20.563302 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:20.566102 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:20.566697 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:20.567361 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:20.569803 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:20.570900 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:20.665318 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:20.699625 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:20.764945 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:20.775558 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:20.776203 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:20.777128 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:20.792609 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:20.793761 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:20.793807 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:20.803515 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:20.810925 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:20.813939 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:20.814229 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:20.838930 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:20.985402 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.010875 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:21.018571 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.033845 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.091166 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.123910 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.138535 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.177318 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.207353 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:21.213932 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:21.217584 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:21.218634 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:21.218632 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 08:01:21.223351 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:21.231024 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:21.241752 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:21.247582 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:21.258695 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:21.322055 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:21.334307 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:21.364383 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:21.419044 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:21.421530 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:21.430236 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:21.433358 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:21.443821 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:21.448740 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:21.449627 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:21.457219 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:21.466984 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.507781 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.559037 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.561831 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.604365 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:21.655876 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:21.655961 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:21.663939 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:21.665882 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:21.671815 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:21.688055 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:21.690598 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:21.726531 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.746522 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.788858 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:21.814946 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.852859 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:21.873919 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:21.888257 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:21.890079 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:21.901379 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:21.904435 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:21.916507 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:21.918800 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:21.936123 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:21.967803 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:22.007169 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:22.025070 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:22.065310 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:22.083781 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:22.114868 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:22.115463 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:22.115763 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:22.116129 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:22.125458 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:22.136977 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:22.172356 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:22.231102 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:22.232028 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:22.233971 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:22.277523 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:22.346565 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:22.349282 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:22.350785 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:22.361191 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:22.381860 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:22.382174 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:22.390537 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:22.410151 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:22.430250 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:22.549829 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:22.608187 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:22.653091 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:22.752257 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:22.771754 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:22.771852 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:22.782717 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:22.793444 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:22.794484 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:22.796126 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:22.801088 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:22.807101 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:22.824041 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:22.826303 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:22.845438 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:22.891031 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:22.927870 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:22.949323 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:22.987969 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:22.999706 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:23.003476 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:01:23.012043 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:23.012893 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:23.025022 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:23.025341 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:23.043712 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:23.056317 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:23.125167 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:23.216246 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:23.217247 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:23.221809 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:23.228648 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:23.235241 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:23.235440 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:23.237924 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:23.254987 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:23.255424 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:23.271890 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:23.284134 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:23.297101 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:23.312583 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:23.375109 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:23.387907 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:23.402098 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:23.453127 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:23.455049 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:23.457432 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:23.457801 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:23.458407 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:23.464787 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:23.472771 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:23.474193 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:23.494519 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:23.507656 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:23.557686 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:23.650743 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:23.660455 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:23.689281 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:23.693579 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:23.694625 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:23.697446 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:23.704114 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:23.704816 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:23.709168 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:01:23.715472 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:23.718603 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:23.727531 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:23.738953 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:23.913859 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:23.918272 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:23.918467 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:23.922495 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:23.925323 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:23.931129 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:23.932500 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:23.967777 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:24.010561 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.020948 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.033037 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.051506 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.111744 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:24.112242 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.117815 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.134711 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:24.208746 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.225185 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.239423 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:24.239576 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.271827 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.306178 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:01:24.346905 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:24.352018 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:24.357544 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:24.358105 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:24.367058 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:24.368613 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:24.396607 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:24.487402 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.530809 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:24.568650 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:24.579021 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:24.583509 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:24.586977 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:24.588107 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:24.588955 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:24.589411 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:24.594532 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:24.604090 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.606185 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:24.786621 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240503 08:01:24.824048 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:24.836975 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:24.841674 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:24.847179 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:24.856480 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:24.867128 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:24.868078 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:24.871718 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:24.874097 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:24.897504 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:01:24.989866 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:25.007201 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:01:25.056461 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:25.062979 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:25.066963 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:01:25.067229 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:25.086755 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:25.086742 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:25.087438 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:25.087857 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:01:25.088589 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:25.124164 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:25.148277 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:25.157467 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 08:01:25.211947 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:25.291947 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:01:25.294019 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:25.296778 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:25.301891 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:25.313517 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:25.323661 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:25.336400 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:25.342475 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:25.345650 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240503 08:01:25.497642 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:01:25.498397 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:01:25.524734 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:25.535022 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:25.535156 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:25.537652 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:25.538529 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:25.541728 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:25.562448 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:25.567725 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:25.579437 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:25.587112 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:25.605342 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:25.643010 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:25.656916 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:25.660006 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:01:25.668848 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:25.670769 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:25.690943 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:25.713088 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:25.794528 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:25.803239 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:25.815589 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:25.878626 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:25.911267 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:25.917979 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:01:25.932448 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:25.938580 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:25.940099 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:25.945317 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:01:25.945756 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:25.952281 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:25.957116 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:25.965445 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:25.974749 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:26.015533 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:26.075125 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:26.117511 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:26.117695 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:01:26.154547 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:26.155501 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:26.162042 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:26.164515 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:26.166234 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:26.171175 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:26.172699 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:26.207893 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:26.231778 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:26.253499 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:26.272550 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:26.283690 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:26.284633 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:26.287346 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:26.346824 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:26.372289 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:26.373291 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:26.386863 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:26.387112 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:26.387257 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:26.396034 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:26.403666 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:26.451472 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:26.511754 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:26.596933 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:26.613096 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:01:26.618212 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:26.619536 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:26.632086 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:26.637506 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:26.642551 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:26.642904 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:26.653326 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:26.657814 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:26.667645 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:26.688789 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:26.688789 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:26.730909 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:26.741712 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling FlushMRSOp(cb2190946fb04a6392097f9588ffda6b): perf score=0.033357
I20240503 08:01:26.761046 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: FlushMRSOp(cb2190946fb04a6392097f9588ffda6b) complete. Timing: real 0.018s user 0.011s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3420,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1577,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":697,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":133}
I20240503 08:01:26.762960 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling UndoDeltaBlockGCOp(cb2190946fb04a6392097f9588ffda6b): 4923 bytes on disk
I20240503 08:01:26.763415 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: UndoDeltaBlockGCOp(cb2190946fb04a6392097f9588ffda6b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240503 08:01:26.845178 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:26.846504 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:26.867813 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:26.873358 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:26.880553 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:26.882583 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:26.887416 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:26.888267 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:26.890605 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:26.947397 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:27.008893 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:27.015254 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling FlushMRSOp(f365955974634bc58e8a0da0fc2563d7): perf score=0.033363
I20240503 08:01:27.016747 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling FlushMRSOp(0fcc4c2eeaec4b359108ec64a6e20da8): perf score=0.033398
W20240503 08:01:27.023685 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:27.023725 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:27.045229 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: FlushMRSOp(f365955974634bc58e8a0da0fc2563d7) complete. Timing: real 0.028s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":5278,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1767,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2070,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":728,"threads_started":2,"wal-append.queue_time_us":1497}
I20240503 08:01:27.047097 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling UndoDeltaBlockGCOp(f365955974634bc58e8a0da0fc2563d7): 4923 bytes on disk
I20240503 08:01:27.045181 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: FlushMRSOp(0fcc4c2eeaec4b359108ec64a6e20da8) complete. Timing: real 0.026s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":641,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":94,"threads_started":1,"wal-append.queue_time_us":1407}
I20240503 08:01:27.047799 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: UndoDeltaBlockGCOp(f365955974634bc58e8a0da0fc2563d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 08:01:27.051904 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling UndoDeltaBlockGCOp(0fcc4c2eeaec4b359108ec64a6e20da8): 4923 bytes on disk
I20240503 08:01:27.054653 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: UndoDeltaBlockGCOp(0fcc4c2eeaec4b359108ec64a6e20da8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240503 08:01:27.103783 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:27.113518 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:27.117286 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:27.117609 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:27.120353 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:27.144527 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:27.170257 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:27.184034 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:27.212193 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:27.227325 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:27.292868 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:27.308410 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:27.410903 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:27.465210 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:27.484158 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:27.532416 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:27.549830 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:27.550557 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:27.557830 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:27.559661 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:27.560175 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:27.564648 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:27.564745 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling FlushMRSOp(ba1de17d15b74c4497e649aadc8569d2): perf score=0.033391
I20240503 08:01:27.567133 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:27.612504 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:27.624112 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: FlushMRSOp(ba1de17d15b74c4497e649aadc8569d2) complete. Timing: real 0.059s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":241,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":5549,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":13576,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":689,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":8516}
I20240503 08:01:27.626035 9814 maintenance_manager.cc:419] P a372e98a670f49e9adca46ab5c836c4a: Scheduling UndoDeltaBlockGCOp(ba1de17d15b74c4497e649aadc8569d2): 4923 bytes on disk
I20240503 08:01:27.626606 9704 maintenance_manager.cc:643] P a372e98a670f49e9adca46ab5c836c4a: UndoDeltaBlockGCOp(ba1de17d15b74c4497e649aadc8569d2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
W20240503 08:01:27.661012 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:27.764809 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:27.770737 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:27.772909 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:27.776129 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:27.780361 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:27.804986 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:27.815212 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:27.819954 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:27.839583 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:27.849848 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:27.968757 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:27.994307 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:28.005379 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:28.008124 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:28.015563 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:28.015947 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:28.018524 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:28.020920 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:28.029228 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:28.045791 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.049823 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.060566 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:28.073271 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.157892 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:28.210420 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.210608 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:28.229034 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:28.232064 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:28.233506 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:28.248639 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:28.252758 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:28.254026 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:28.256982 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:28.256966 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:28.264130 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:28.264897 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:28.308133 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:28.308897 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:28.323297 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:28.362066 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:28.394537 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.397218 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:28.480440 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:28.484301 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:28.491451 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:28.490931 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:28.493269 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:28.502526 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:28.514569 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:28.552862 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:28.587283 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:28.604473 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.633116 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:28.640019 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.657176 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.699448 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:28.723536 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:28.732560 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:28.735586 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:28.740110 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:28.744155 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:28.759037 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:28.777686 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:28.845909 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:28.875412 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.942274 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.942934 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.963229 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.967988 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:28.991856 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:29.114574 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:29.146996 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:01:29.151724 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:29.152166 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:29.153111 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:29.160203 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:29.166519 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:29.170805 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:29.174887 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:29.187718 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:29.190295 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:29.216367 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:29.241231 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:29.317791 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:29.322243 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:29.339692 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:29.378239 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:29.385332 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:29.387939 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:29.388407 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:29.390466 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:29.390533 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:29.419688 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:29.538308 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:29.566619 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:29.603188 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:29.615473 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:29.617964 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:29.618613 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:29.619093 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:29.620338 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:29.622202 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:29.625741 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:29.636698 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:29.680505 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:29.709026 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:29.743158 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:29.747413 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:29.785391 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:29.820761 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 104 similar messages]
I20240503 08:01:29.831745 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:29.833818 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:29.840775 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:29.841149 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:29.842816 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:29.843360 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:29.875145 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:29.904523 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 08:01:29.920665 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:01:29.971043 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:29.986390 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:30.063443 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:30.070222 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:30.071617 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:30.095940 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:30.096055 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:30.101981 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:30.106053 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:30.118325 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:30.121877 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:30.137703 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:30.184084 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 08:01:30.225473 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:30.300189 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:30.300475 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:30.302664 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:30.304839 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:30.307307 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:30.325342 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:30.344309 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:30.344854 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:30.347056 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:30.389850 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:30.425328 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 08:01:30.505344 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:01:30.506048 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:30.509343 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:30.531351 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:30.559324 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:30.632021 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:30.635744 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:30.644939 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:30.654814 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:30.699657 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:30.721158 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:30.725031 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:01:30.728722 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:30.730191 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:30.730705 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:30.740995 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:30.743479 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:30.746651 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:30.750530 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:30.778450 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:30.779207 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:30.779347 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:30.779558 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:30.802927 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:30.804533 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:30.811268 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:30.851440 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:30.918246 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:30.948881 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:30.950562 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:30.964780 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:30.964872 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:30.972945 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:30.974521 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:30.982903 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:31.013370 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:31.171588 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:31.182497 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:31.184549 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:31.186606 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:31.191779 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:31.203434 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:31.209120 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:31.209180 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:31.218392 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:31.220033 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:31.255227 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:31.381042 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:31.415788 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:31.421782 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:31.432248 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:01:31.433069 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:31.439074 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:31.443913 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:31.444392 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:31.457871 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:31.462930 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:31.465318 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:31.497817 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:31.524760 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:31.549105 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:31.549896 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:31.562306 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:31.635099 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:31.659021 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:31.660948 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:31.662117 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:31.666332 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:31.666332 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:31.669350 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:31.673715 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:31.673998 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:31.680186 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:31.692803 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:31.771437 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:31.803208 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:31.828294 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:31.890074 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:31.894289 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:31.902225 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:31.902617 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:31.905483 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:31.914592 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:31.925580 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:31.925906 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:32.036432 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:32.040230 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:32.060359 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:32.095919 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:01:32.110512 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:32.115180 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:32.318634 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:32.324885 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:32.328608 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:32.332211 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:32.333030 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:32.340473 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:32.345502 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:32.347736 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:32.413443 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:32.430491 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:32.451184 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:32.506606 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:01:32.533398 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:32.533988 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:32.537125 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:32.547792 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:32.549839 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:32.561591 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:32.567489 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:32.582154 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:32.727977 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:32.735316 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:32.748893 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:32.751204 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:32.785239 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:32.785714 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:32.786741 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:32.791450 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:32.791488 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:32.793569 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:32.797014 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:32.808313 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:32.848914 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling FlushMRSOp(64d33ff4164b49a7948aa5afee40bf3f): perf score=0.033384
W20240503 08:01:32.856528 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:32.860390 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:32.868045 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: FlushMRSOp(64d33ff4164b49a7948aa5afee40bf3f) complete. Timing: real 0.019s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":2036,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":675,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":121}
I20240503 08:01:32.870694 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling UndoDeltaBlockGCOp(64d33ff4164b49a7948aa5afee40bf3f): 4581 bytes on disk
I20240503 08:01:32.871068 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: UndoDeltaBlockGCOp(64d33ff4164b49a7948aa5afee40bf3f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
W20240503 08:01:32.877022 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:32.910949 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:32.911921 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:32.989569 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:32.993249 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:33.004372 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:33.012137 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:33.012389 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:33.012902 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:33.015585 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:33.036157 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:33.063179 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:33.074853 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:33.139163 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:33.182734 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:33.220898 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:33.226207 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:33.231496 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:33.243748 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:33.245302 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:33.245649 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:33.259090 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:33.280817 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:33.295583 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:33.309367 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:33.315456 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:33.337432 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:33.414098 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:33.420858 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:33.445572 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:33.463114 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:33.470037 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:33.480990 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:33.483882 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:33.489627 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:33.494511 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:33.519577 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:33.531594 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:33.553557 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:33.624506 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:33.644495 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:33.653843 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:33.672545 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:33.688428 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:33.723351 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:33.767534 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:33.832047 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:33.885046 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:33.888489 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:33.890834 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:33.900068 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:33.905920 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:33.914507 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:33.917656 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:34.014246 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:34.021427 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:34.021682 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:34.103467 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:34.108129 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:34.110713 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:34.115221 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:34.115762 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:34.129738 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:34.132177 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:34.133644 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:34.138301 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:34.145365 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:34.251443 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:34.269147 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:34.274086 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:34.295797 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:34.306831 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:34.314532 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:34.343657 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:34.344707 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:34.346720 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:34.347079 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:34.347704 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:34.355392 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:34.368206 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:34.398681 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:34.543962 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:34.567909 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:34.572227 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:34.573691 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:34.574693 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:34.583959 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:34.615362 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:34.615890 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:34.631481 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:34.650535 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:34.733207 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:34.738565 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:01:34.748260 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:34.756078 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:34.804522 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:34.811573 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:34.819795 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:34.819802 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:34.831004 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:34.835498 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:34.856169 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:34.860870 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:34.887678 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240503 08:01:34.949086 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:34.980624 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:01:35.020371 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:35.048839 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:35.052899 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:35.055552 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:35.058499 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:35.061375 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:35.066542 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:35.080482 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:35.083756 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:35.095139 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:35.113452 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:35.117523 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling FlushMRSOp(5390e70b0449449aa3af05112bd29797): perf score=0.033367
W20240503 08:01:35.132792 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:01:35.133477 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: FlushMRSOp(5390e70b0449449aa3af05112bd29797) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":172,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":2691,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":509,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":1792,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":139}
I20240503 08:01:35.135341 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling UndoDeltaBlockGCOp(5390e70b0449449aa3af05112bd29797): 4583 bytes on disk
I20240503 08:01:35.135748 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: UndoDeltaBlockGCOp(5390e70b0449449aa3af05112bd29797) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240503 08:01:35.154213 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:35.238214 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 08:01:35.238860 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:35.374715 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:35.390743 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling FlushMRSOp(7bbaf13cbb474734a961664ae8ebe779): perf score=0.033373
W20240503 08:01:35.397877 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:35.406451 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: FlushMRSOp(7bbaf13cbb474734a961664ae8ebe779) complete. Timing: real 0.015s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":140,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1610,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":556,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":122,"threads_started":2,"wal-append.queue_time_us":121}
I20240503 08:01:35.408463 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling UndoDeltaBlockGCOp(7bbaf13cbb474734a961664ae8ebe779): 4923 bytes on disk
I20240503 08:01:35.408969 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: UndoDeltaBlockGCOp(7bbaf13cbb474734a961664ae8ebe779) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 08:01:35.421342 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:35.447204 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:35.447363 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:35.452931 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:35.492717 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:35.494524 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:35.499737 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:35.503501 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:35.513331 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:35.519932 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:01:35.523430 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:35.531461 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:35.534518 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 101 similar messages]
I20240503 08:01:35.541728 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:35.563814 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:35.580611 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:35.600055 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:35.632408 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:35.663939 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:35.702984 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:35.712972 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:35.715672 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:35.718096 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:35.718331 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:35.727399 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:35.733292 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:35.738863 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:35.765983 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:35.772481 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:01:35.772805 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:35.798739 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:35.833206 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:35.883133 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:35.912124 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:01:35.923089 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:35.924612 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:35.937443 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:35.947428 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:35.952318 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:35.959563 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:35.964149 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:35.988739 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:36.042047 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.090904 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.108649 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:01:36.149376 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:36.149783 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:36.155043 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:36.158579 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:36.160600 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:36.164273 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:36.182824 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:36.193189 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:36.200115 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:36.217734 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.238675 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.307611 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:36.378751 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:36.382607 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:36.389154 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:36.397789 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:36.400506 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:36.401162 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:36.402931 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:36.406921 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:36.569128 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:01:36.584698 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.585592 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.603410 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.614475 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:01:36.617223 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:36.624027 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:36.624601 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:36.628954 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:36.629127 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:36.632804 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:36.680150 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:36.717967 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.718331 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.745709 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.815559 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:36.833171 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:36.838727 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:36.861479 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.879658 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.886400 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:36.919165 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:37.036718 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:01:37.057864 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:37.058274 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:37.070612 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:37.071985 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:37.078289 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:37.079632 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:37.082710 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:37.096580 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:37.272850 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:37.288528 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:37.288844 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:37.288935 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:37.289376 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:37.290602 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:37.311156 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:37.318629 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:37.320246 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:37.332657 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:01:37.381882 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:37.441638 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:37.477656 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:01:37.513242 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:37.515341 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:37.522392 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:37.522768 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:37.526602 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:37.533744 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:01:37.540496 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:37.547821 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:01:37.570577 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:37.713583 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:37.733383 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:37.737758 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:37.737880 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:37.742233 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:37.743970 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:37.747642 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:37.771098 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:37.781360 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:37.803333 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:37.810477 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:37.818075 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:37.886459 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:37.926106 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:37.933808 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:37.943991 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:01:37.958308 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:37.963383 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:37.964654 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:37.970487 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:37.974148 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:37.991997 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:38.004217 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:01:38.006609 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:38.036180 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:38.041389 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:38.119051 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:38.123957 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:38.151224 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:38.157023 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:01:38.180526 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:38.182600 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:38.185024 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:38.185140 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:38.212011 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:38.219437 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:38.221984 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:38.232177 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:38.291589 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:38.294426 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:38.297271 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:38.297325 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:38.309104 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:38.309299 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:38.310428 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:38.334733 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:38.458945 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:38.473233 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:38.548511 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:01:38.597023 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:38.597292 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:38.601368 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:01:38.609499 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:38.609522 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:38.611793 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:38.619338 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:38.647398 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:38.779973 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:01:38.823901 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:38.826041 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:38.827181 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:38.828955 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:38.832293 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:38.858914 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:38.859351 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:38.872104 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:38.872924 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:38.893407 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:38.970592 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:39.026779 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:01:39.046777 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:39.050027 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:39.055562 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:39.060245 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:39.068869 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:39.074237 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:39.084241 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:39.169677 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:39.186486 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:39.239178 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:01:39.265040 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:39.268975 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:39.276034 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:01:39.279376 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:39.286890 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:39.298924 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:39.306969 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:39.314635 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:01:39.319432 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:39.332692 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:39.334264 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:39.353847 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:39.389123 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:39.423393 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:39.465735 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:39.467764 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:01:39.506165 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:39.511518 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:39.512552 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:39.518957 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:39.519094 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:39.521368 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:39.522807 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:01:39.558256 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:39.568648 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:39.597867 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:01:39.723654 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:39.733362 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:39.733695 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:39.734418 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:39.751264 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:39.755554 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:39.756981 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:39.777514 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:39.915119 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:39.934491 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240503 08:01:39.935636 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:39.988093 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 08:01:40.044008 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:01:40.046238 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.076968 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:01:40.140612 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:40.141974 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:40.144907 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:40.145349 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:40.145577 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:40.150604 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:01:40.151903 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:40.158834 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:40.159046 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:40.164028 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.241822 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 08:01:40.261718 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.289584 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:40.350503 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:40.350503 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:40.354063 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:40.363121 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:40.363791 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:40.374506 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:40.375727 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:40.381829 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:40.393237 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.409538 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.447331 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.463879 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.516062 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.520262 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.536746 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 08:01:40.555424 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:40.555462 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:01:40.573668 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:40.577656 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:40.580412 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:40.586675 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:40.587272 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:40.588801 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:40.592465 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:01:40.594139 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.598083 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:40.604318 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:40.614706 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:40.622589 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.678637 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:40.704524 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.708827 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.731113 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:40.751235 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.760509 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:40.804747 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:40.810766 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:40.813118 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:40.829322 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:40.834684 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:40.839064 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:40.846608 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:40.873207 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:01:40.915434 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.951494 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:40.960163 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:40.991950 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:41.026530 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:01:41.039882 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:41.040001 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:41.042553 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:41.047030 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:41.049295 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:41.051713 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:41.056933 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:41.077946 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:41.082082 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:41.161073 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:41.218842 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:41.265260 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:41.273231 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:41.279968 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:41.286907 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:41.294888 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:41.301324 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:41.316424 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:41.416226 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:41.551466 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:41.568037 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:41.601892 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:01:41.604233 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:41.621018 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:41.624842 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:41.633531 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:41.692813 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:41.693812 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:41.697126 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:41.703702 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:41.706292 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:41.711266 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:41.723708 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:41.723881 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:41.727231 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:41.731926 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:41.749262 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:41.774245 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:41.851483 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:41.898013 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:41.900418 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:41.907315 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:41.924526 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:41.926199 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:41.928035 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:41.929903 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:42.003556 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:42.013844 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:42.050057 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:42.082612 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:42.123289 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:42.125842 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:42.126534 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:42.130699 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:42.131124 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:42.133970 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:42.134519 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:42.165935 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:42.174752 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:42.258001 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:42.298018 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:42.347738 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:42.349013 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:42.349375 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:42.350962 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:42.352797 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:42.352967 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:42.379478 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:42.438174 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:42.541482 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:42.544865 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:42.566404 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:42.569604 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:42.575017 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:42.582451 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:42.583243 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:42.590795 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:42.600045 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:42.604698 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:42.646708 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:42.652081 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:42.652432 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:42.691421 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:42.717449 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:42.722652 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:01:42.768422 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:42.797319 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:42.801015 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:42.801898 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:42.812527 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:42.820113 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:42.823909 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:42.846267 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:42.887741 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:42.912503 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:42.918517 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:42.934595 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:43.052469 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:43.061841 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:43.135043 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:43.159914 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:43.174089 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:43.178870 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:43.215013 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:43.217224 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:43.226469 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:43.231546 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:43.232244 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:43.232251 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:43.242453 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:43.256855 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:43.262982 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:43.275642 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:43.284020 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:43.286847 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:43.298370 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:43.307098 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:43.349174 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:43.377661 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:43.386852 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:43.442849 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling FlushMRSOp(31c76213f3ec451084ae7b0551cd7b48): perf score=0.033355
I20240503 08:01:43.467165 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:43.468456 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:43.471935 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:43.476162 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:43.495735 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:43.512166 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: FlushMRSOp(31c76213f3ec451084ae7b0551cd7b48) complete. Timing: real 0.069s user 0.018s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1808,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":177,"threads_started":2,"wal-append.queue_time_us":211}
I20240503 08:01:43.514657 7198 maintenance_manager.cc:419] P 00a3f8e159c64327832d79fb32cdcca8: Scheduling UndoDeltaBlockGCOp(31c76213f3ec451084ae7b0551cd7b48): 4923 bytes on disk
I20240503 08:01:43.515080 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:43.516577 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:43.518604 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:43.521474 7088 maintenance_manager.cc:643] P 00a3f8e159c64327832d79fb32cdcca8: UndoDeltaBlockGCOp(31c76213f3ec451084ae7b0551cd7b48) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240503 08:01:43.552882 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:43.557683 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:43.656289 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:43.683782 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:43.696906 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling FlushMRSOp(16c06ab5a3e54ed2910afa50069810cc): perf score=0.033398
W20240503 08:01:43.710561 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:43.718753 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:43.726972 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:43.734323 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:43.741109 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:43.741533 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:43.755875 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:43.761476 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:43.769372 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:43.805985 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: FlushMRSOp(16c06ab5a3e54ed2910afa50069810cc) complete. Timing: real 0.109s user 0.017s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":221,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":1906,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":632,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":202,"threads_started":2,"wal-append.queue_time_us":27239}
I20240503 08:01:43.810236 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling FlushMRSOp(660355a4482d4a30bbb71cd0b56527d1): perf score=0.033382
I20240503 08:01:43.830226 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: FlushMRSOp(660355a4482d4a30bbb71cd0b56527d1) complete. Timing: real 0.020s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1398,"drs_written":1,"lbm_read_time_us":148,"lbm_reads_lt_1ms":4,"lbm_write_time_us":628,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":93,"threads_started":1,"wal-append.queue_time_us":220}
I20240503 08:01:43.832540 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling UndoDeltaBlockGCOp(16c06ab5a3e54ed2910afa50069810cc): 4838 bytes on disk
I20240503 08:01:43.833422 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: UndoDeltaBlockGCOp(16c06ab5a3e54ed2910afa50069810cc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 08:01:43.835276 9552 maintenance_manager.cc:419] P cb73994515af4880b05383e78368bb4a: Scheduling UndoDeltaBlockGCOp(660355a4482d4a30bbb71cd0b56527d1): 4581 bytes on disk
I20240503 08:01:43.835700 9442 maintenance_manager.cc:643] P cb73994515af4880b05383e78368bb4a: UndoDeltaBlockGCOp(660355a4482d4a30bbb71cd0b56527d1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 08:01:43.980289 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:43.982733 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:43.983314 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:43.986411 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:43.991806 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:44.019483 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:44.022950 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:44.030470 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:44.054024 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:44.077421 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:44.125422 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:44.150228 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:44.153553 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:44.197427 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:44.209582 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:44.209956 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:44.210014 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:44.215212 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:44.223290 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:44.236953 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:44.270859 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:44.305424 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:44.308620 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:44.317831 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:44.363412 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:44.376029 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:44.383319 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:44.399924 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:44.408763 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:44.457711 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:44.460925 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:44.461759 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:44.474445 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:44.493359 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:44.506398 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:44.507855 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:44.520784 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:44.583180 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:44.622495 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:44.639134 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:44.650943 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:44.755517 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:44.756915 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:44.854739 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:44.883669 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:44.885941 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:44.890161 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:44.904888 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:44.911418 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:44.931143 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:44.931865 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:44.978294 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240503 08:01:44.992566 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 08:01:45.063992 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:01:45.117704 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:45.118728 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:45.119604 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:45.122398 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:45.125948 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:45.126760 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:45.132479 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:45.136140 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.238757 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:45.254854 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240503 08:01:45.255820 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.301007 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.323369 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:01:45.324849 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:45.340865 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:45.341923 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:45.345084 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:45.346474 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:45.353145 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:45.353425 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:45.386560 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:45.398957 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.409106 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.431627 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.459832 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.506882 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:45.511345 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:45.560977 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240503 08:01:45.567101 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:45.572712 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:45.574697 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:45.577934 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:45.581471 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:45.585450 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:45.585721 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:45.588678 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:45.609618 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:01:45.613205 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:45.616225 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:45.628815 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:45.629108 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.642958 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:45.676327 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.686082 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:45.693781 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.732458 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:45.802801 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:45.807468 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:45.817950 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:45.820999 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:45.823705 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:45.824613 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:45.830555 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:45.837989 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:45.838511 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:45.850334 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:45.864419 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.869386 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:45.900091 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:01:45.900848 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:45.904958 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.960860 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:45.985527 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:46.052277 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:46.060959 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:46.062157 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:46.077313 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:46.082168 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:46.083777 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:46.089161 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:46.115026 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:46.146497 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:46.172534 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:46.439338 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:46.451817 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:46.492168 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:46.493232 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:46.497618 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:46.520380 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:46.528344 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:46.531188 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:46.544283 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:46.563268 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:46.597187 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:46.648993 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:46.696666 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:46.697638 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:46.712663 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:46.714963 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:46.722153 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:46.722465 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:46.732856 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:46.738574 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:46.739601 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:46.758455 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:46.759970 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:46.767084 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:46.870492 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:46.874166 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:46.881767 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:46.892066 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:46.934674 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:46.957129 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:46.957832 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:46.968149 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:46.977129 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:46.981151 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:46.981657 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:46.981747 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:46.996552 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:47.001221 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:47.030790 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:47.045820 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:47.063535 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:47.190786 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:47.192934 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:47.205103 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:47.205315 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:47.209003 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:47.212007 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:47.212738 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:47.222739 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:47.230544 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:47.242981 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:47.243664 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:47.425320 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:47.429822 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:47.432096 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:47.432498 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:47.438918 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:47.460927 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:47.472457 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:47.554733 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:47.580212 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:01:47.649478 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:47.657001 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:47.664172 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:47.665608 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:47.669888 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:47.673743 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:47.674324 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:47.675240 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:47.675671 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:47.763870 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:47.772749 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:47.800678 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:47.809463 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:47.841248 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:01:47.946481 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:47.950197 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:47.987723 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:48.026156 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:48.032799 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:48.043342 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:48.067317 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:48.079790 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:48.086685 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:48.088061 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:48.090883 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:48.098762 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:48.104241 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:48.104615 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:48.120398 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:48.128893 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:48.146667 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:48.205049 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:48.224874 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:48.252463 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:48.261456 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:48.262180 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:48.286129 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:48.294790 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:48.300804 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:48.303001 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:48.305519 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:48.313503 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:48.321322 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:48.323493 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:48.346879 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:48.383925 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:48.421847 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:48.438324 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:48.455979 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:48.504058 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:48.511027 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:48.517354 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:48.524701 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:48.533869 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:48.534171 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:48.535053 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:48.540972 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:48.543661 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:48.556919 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:48.568876 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:48.728785 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:48.737428 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:48.745083 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:48.745777 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:48.751292 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:48.753540 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:48.780629 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling FlushMRSOp(1ae7895f7d7f47c3ab5fb0b93c1fd8e0): perf score=0.033385
I20240503 08:01:48.791647 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:48.794028 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:48.805150 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: FlushMRSOp(1ae7895f7d7f47c3ab5fb0b93c1fd8e0) complete. Timing: real 0.024s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":186,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1033,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":598,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":24960,"thread_start_us":165,"threads_started":2,"wal-append.queue_time_us":176}
I20240503 08:01:48.807263 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling FlushMRSOp(5b794508aa77492398e48d25edfb8aa3): perf score=0.033342
I20240503 08:01:48.825524 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: FlushMRSOp(5b794508aa77492398e48d25edfb8aa3) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1339,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":489,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":80,"threads_started":1,"wal-append.queue_time_us":175}
I20240503 08:01:48.827579 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling UndoDeltaBlockGCOp(1ae7895f7d7f47c3ab5fb0b93c1fd8e0): 4583 bytes on disk
I20240503 08:01:48.828258 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: UndoDeltaBlockGCOp(1ae7895f7d7f47c3ab5fb0b93c1fd8e0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 08:01:48.829955 9683 maintenance_manager.cc:419] P 159177b3caeb434baa98e04136d050b2: Scheduling UndoDeltaBlockGCOp(5b794508aa77492398e48d25edfb8aa3): 4583 bytes on disk
I20240503 08:01:48.830446 9573 maintenance_manager.cc:643] P 159177b3caeb434baa98e04136d050b2: UndoDeltaBlockGCOp(5b794508aa77492398e48d25edfb8aa3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240503 08:01:48.952749 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:48.971586 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:48.973119 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:48.992825 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:48.994947 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:48.997242 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:49.004164 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:49.013414 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:49.014894 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:49.014972 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:49.026878 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:49.119174 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:49.149725 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:49.160689 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:01:49.190848 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:49.195902 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:49.196455 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:49.205499 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:49.208658 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:49.216609 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:49.227519 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:49.231978 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:49.233738 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:49.266712 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:49.296108 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:49.297907 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:49.444272 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:49.448295 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:49.492362 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:49.537307 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:49.542812 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:49.594103 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:49.605566 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:49.612386 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:49.614984 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:49.615840 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:49.626684 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:49.643538 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:49.672427 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:49.717551 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:49.792809 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:49.830739 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:49.835304 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:49.839411 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:49.842125 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:49.851125 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:49.851686 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:49.862720 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:49.864118 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:49.865445 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:49.995471 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.005643 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240503 08:01:50.065222 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:50.068593 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:50.069684 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 101 similar messages]
I20240503 08:01:50.073089 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:50.074229 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:50.089814 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:50.090664 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:50.108070 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:50.134505 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.143126 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.165935 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:50.247138 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.275772 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:50.277436 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:50.282572 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:50.283723 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:50.294737 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:50.295328 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:50.299203 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:50.309618 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:50.313601 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:50.363065 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.388034 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240503 08:01:50.391131 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.420938 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.457340 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:01:50.480820 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.500551 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:50.508551 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:50.509974 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:50.509994 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:50.521342 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:50.521904 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:50.528066 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:50.531997 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:50.552059 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:50.552767 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:50.591429 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 08:01:50.650318 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.652993 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:50.691094 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.712771 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:01:50.714653 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:50.728996 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:50.731479 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:50.735863 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:50.737437 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:50.738531 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:50.740289 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:50.743909 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:50.746104 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:50.764493 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:50.809521 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.809595 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.823348 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:50.846179 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.872538 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:50.888391 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:50.898485 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:50.921466 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:01:50.941946 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:51.012921 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:51.094544 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:51.116243 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:51.142053 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:51.143999 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:51.149138 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:51.156406 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:51.157367 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:51.157888 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:51.158967 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:51.184511 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:51.195660 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:51.200879 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:51.254112 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:51.359416 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:51.372905 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:51.373183 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:51.375576 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:51.388068 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:51.388756 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:51.388883 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:51.396525 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:51.469405 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:51.480809 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:51.495668 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:51.582700 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:51.585821 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:51.586946 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:51.592289 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:51.593856 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:51.594450 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:51.608016 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:51.619249 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:51.622264 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:51.692858 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:51.729274 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:51.792019 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:01:51.795289 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:51.796996 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:51.800745 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:51.801745 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:51.810519 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:51.810997 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:51.819339 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:51.847352 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:51.857599 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:51.865814 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:51.902117 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:51.966735 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:51.977182 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:51.991765 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:52.017747 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:52.028553 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:52.028827 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:52.028837 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:52.038511 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:52.043999 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:52.054244 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:52.056502 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:52.146613 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:52.179396 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:52.227520 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:52.237957 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:52.240036 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:52.241295 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:52.244241 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:52.246259 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:52.260710 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:52.264559 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:52.296578 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:52.369421 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:52.425438 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:52.469748 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:52.557924 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:52.561096 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:52.655022 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:52.657370 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:52.667852 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:52.668499 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:52.691879 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:52.696549 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:52.710207 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:01:52.719066 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:52.801087 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:52.807142 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:52.866170 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:52.872864 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:52.874807 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:52.875638 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:52.884093 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:52.888963 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:52.920778 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:52.922734 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:53.006536 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:53.025492 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:53.035774 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:53.037317 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:53.065683 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:53.075446 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:53.102766 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:53.105695 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:53.113010 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:53.116616 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:53.118492 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:53.122571 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:53.125339 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:53.126313 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:53.129598 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:53.140321 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:53.140637 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:53.149096 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:53.156502 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:53.185930 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:01:53.194725 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:53.198709 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:53.227703 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:53.274124 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:53.307714 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:53.313657 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:53.317597 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:53.319201 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:53.329660 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:53.330719 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:53.340246 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:53.348042 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:53.377727 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:53.380802 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:53.475622 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:53.479455 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:53.485155 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:53.487790 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:53.520444 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:53.545079 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:53.548378 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:53.551646 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:53.562034 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:53.563077 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:53.567673 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:53.570122 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:53.576714 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:53.584509 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:53.600850 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:53.645430 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:53.668123 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:53.754276 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:53.762892 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:53.764629 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:53.764848 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:53.777699 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:53.779700 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:53.783926 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:53.806015 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:53.868950 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:53.933857 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:54.013029 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:54.076305 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:54.131275 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:54.185403 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:54.192080 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:54.195997 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:54.199442 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:54.207755 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:54.208501 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:54.220528 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:54.240303 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:54.291800 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:54.323505 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:54.325234 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:54.341111 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:54.374956 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:54.402511 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:54.412137 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:54.413223 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:54.414294 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:54.430240 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:54.437412 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:54.438009 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:54.537683 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:54.583560 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:54.599988 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:54.619791 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:54.624692 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:54.626397 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:54.634986 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:54.635828 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:54.637210 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:54.667654 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:54.685638 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:54.715096 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:54.783670 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:54.814322 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:54.824213 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:54.848551 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:54.859380 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:54.862409 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:54.862890 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:54.865486 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:54.875064 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:54.881139 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:54.884112 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:54.942561 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:54.944365 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:54.953094 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:01:55.039657 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 08:01:55.043606 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:55.074636 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:55.085512 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:55.086436 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:55.087074 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:55.088934 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240503 08:01:55.089569 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:55.089709 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:55.092476 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:55.104176 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:55.189062 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:55.202626 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:55.251629 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:55.304129 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:55.305609 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:55.308859 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:55.325482 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:55.325815 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:55.334504 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:55.334564 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:55.397853 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240503 08:01:55.418162 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:55.427843 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:55.439071 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:55.487068 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:55.571837 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:55.577965 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:55.628862 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:01:55.630234 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:55.637163 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:55.645625 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:55.648401 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:55.672895 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:55.677336 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:01:55.714109 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:55.721091 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:55.721478 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:55.723435 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:55.723917 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:55.725916 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:55.735133 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:55.755082 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:55.771024 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:55.781661 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:01:55.819396 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:55.833904 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:55.848660 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:55.897079 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:55.924455 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:55.933177 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:55.935217 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:55.935923 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:55.944504 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:01:55.945984 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:55.947118 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:01:55.948593 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:55.959473 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:55.959520 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:55.959784 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:55.966046 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:55.979475 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:55.985440 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:55.986003 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:56.011831 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:56.020151 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:56.088915 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:56.150765 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:56.168768 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:56.173408 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:56.180636 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:56.182534 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:56.183776 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:56.186668 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:56.219710 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:56.223604 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:56.288067 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:01:56.415467 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:56.415833 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:56.416262 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:56.419878 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:56.421519 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:56.421818 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:56.427768 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:56.472071 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:56.508464 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:56.594208 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:56.638083 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:56.641418 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:56.645255 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:56.646582 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:01:56.652909 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:56.658962 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:56.674531 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:56.676155 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:56.692773 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:56.720610 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:56.756315 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:56.779273 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:56.811515 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:56.880237 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:01:56.885458 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:56.888243 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:56.891009 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:56.894951 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:56.895869 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:56.896077 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:56.908579 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:56.915446 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 08:01:56.917847 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:56.952642 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:57.125627 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:57.141799 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:57.142855 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:57.219784 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:57.280699 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:57.316465 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:57.328089 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:57.329916 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:57.330643 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:57.330693 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:57.330987 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:57.344684 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:57.413408 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:57.444667 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:57.468834 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:57.477738 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:57.477851 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:57.518328 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:01:57.545840 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:57.552037 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:57.552317 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:01:57.557437 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:57.560644 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:57.575564 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:57.575855 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:57.583097 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:57.586733 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:57.618896 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:57.672060 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:57.715021 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:01:57.763562 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:57.767503 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:57.769254 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:57.769686 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:57.784063 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:01:57.784718 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:57.790508 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:57.823294 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:57.834919 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:57.856827 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:57.902307 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:57.971558 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:57.991289 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:57.991855 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:57.997150 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:58.006660 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:58.007460 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:58.019081 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:58.035712 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:58.052556 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:58.116009 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:58.174327 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:58.209522 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:58.223806 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:58.233438 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:58.236725 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:58.242063 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:58.243983 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:58.244462 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:58.245414 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:58.247870 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:58.271060 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:01:58.273124 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:58.280678 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:58.298662 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:58.321254 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:58.359491 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:58.404436 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:01:58.430316 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:58.437669 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:58.443214 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:58.444303 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:58.462854 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:58.465914 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:58.477466 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:58.478318 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:58.478435 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:01:58.493115 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:58.493492 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:58.512149 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:58.521042 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:58.565124 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:58.595405 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:58.629061 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:58.644207 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:58.665344 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:58.855033 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:58.870262 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:01:58.870424 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:58.871613 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:58.872467 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:58.874627 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:58.875577 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:58.901254 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:58.907742 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:58.948858 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:58.964713 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:59.062427 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:59.078138 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:01:59.082762 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:59.094399 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:59.099311 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:59.100178 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:59.106946 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:59.115972 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:59.121170 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:59.121802 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:01:59.135426 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:59.154397 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:59.185775 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:59.232964 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:59.323751 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:01:59.326141 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:59.326510 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:59.327179 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:59.330523 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:59.339252 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:59.342540 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:59.374893 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:01:59.390308 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:01:59.493778 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:01:59.524631 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:59.553814 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:01:59.559995 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:59.560889 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:01:59.570781 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:59.571436 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:59.571722 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:59.578029 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:01:59.606062 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:59.650557 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:59.672981 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:01:59.776216 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:01:59.782749 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:59.789629 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:01:59.790905 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:01:59.799851 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:01:59.811285 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:01:59.811921 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:01:59.831216 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:01:59.877130 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:59.916074 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:59.964643 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:01:59.986306 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:00.009264 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:00.009522 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:00.013118 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:00.024755 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:00.030884 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:00.031369 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:00.052930 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240503 08:02:00.056182 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:00.091928 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 08:02:00.092818 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:00.111349 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:00.112926 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:00.188503 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:00.256469 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:00.263109 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:00.277439 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:00.345504 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:00.382920 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:00.428500 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:00.438613 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:00.442499 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:00.443310 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:00.443531 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:00.450331 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:00.455439 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:00.455490 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 08:02:00.456586 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:00.467109 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:00.479332 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:00.589995 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:00.599105 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:00.633358 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:00.636528 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:00.641716 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:00.644145 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240503 08:02:00.644912 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:00.645503 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:00.653446 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:00.654490 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:00.660928 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:00.661521 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:00.661967 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:00.689879 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:00.696880 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:00.735229 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:00.761358 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:00.770326 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:00.771852 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:00.774506 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:00.793462 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:00.804733 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:00.812841 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:00.831008 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:00.857406 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 08:02:00.880807 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:00.887765 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:00.889166 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:00.889590 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:00.892125 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:00.898757 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:00.909368 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:00.933270 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:00.949822 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:00.958781 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:02:00.959566 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:01.084441 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:01.106560 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:01.107079 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:01.112514 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:01.114847 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:01.130008 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:01.137269 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:01.138459 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:01.140914 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:01.147387 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:01.273200 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:01.280802 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:01.282671 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:01.341750 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:01.343461 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:01.344435 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:01.345901 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:01.347854 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:01.359025 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:01.383853 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:01.388367 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:01.500519 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:01.565008 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:01.570013 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:01.570209 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:01.572528 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:01.573817 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:01.584836 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:01.585103 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:01.597896 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:01.615511 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:01.621220 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:01.657869 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:01.759073 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:01.857353 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:01.933751 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:01.986395 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:01.989732 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:01.991163 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:01.993713 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:01.998162 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:02:02.004719 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:02.008898 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:02.012722 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:02.018014 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:02.032969 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:02.044631 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:02.055457 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:02.171471 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:02.203109 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:02.206995 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:02.217464 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:02.222429 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:02.224177 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:02.228250 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:02.241756 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:02.253824 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:02.265182 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:02.283591 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:02.318710 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:02.328379 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:02.413136 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:02.415781 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:02.421967 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:02.428309 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:02.428436 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:02.441301 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:02.479413 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:02.497629 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:02.555572 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:02.561563 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:02.595873 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:02.632061 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:02.635524 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:02.640125 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:02.640759 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:02.640900 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:02.643007 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:02.659695 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:02.674795 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:02.676422 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:02.688885 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:02.862737 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:02.865113 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:02.871644 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:02.875803 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:02.879703 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:02.883220 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:02.883913 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:02.885466 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:02.888348 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:02.912791 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:02.918777 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:02.919569 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:02.941324 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:02.984628 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:02.995947 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:03.000483 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:03.021453 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:03.032487 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:03.090660 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:03.092093 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:03.100540 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:03.104542 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:03.105715 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:03.106523 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:03.113788 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:03.133059 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:03.139541 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:03.146807 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:03.152421 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:03.159472 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:03.160768 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:03.210422 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:03.227955 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:03.242401 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:03.324047 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:03.351397 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:03.395629 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:03.400552 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:03.480456 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:03.516860 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:03.525128 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:03.528441 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:03.534492 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:03.536762 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:03.538394 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:03.539355 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:03.545022 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:03.566725 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:03.590296 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:03.642510 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:03.676977 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:03.746385 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:03.755694 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:03.760476 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:03.766323 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:03.770807 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:03.777681 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:03.796851 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:03.799229 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:03.900578 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:03.902181 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:03.968328 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:03.968478 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:03.972501 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:03.985994 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:03.986104 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:03.987179 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:03.992276 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:03.992576 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:04.019981 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:04.057696 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:04.194705 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:04.197410 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:04.198467 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:04.200456 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:04.202548 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:04.212543 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:04.217620 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:04.238097 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:04.238163 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:04.240737 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:04.322013 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:04.394043 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:04.411226 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:04.415143 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:04.417340 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:04.421697 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:04.430087 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:04.440868 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:04.442118 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:04.466660 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:04.499433 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:04.529426 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:04.549930 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:04.641611 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:04.641762 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:04.643915 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:04.650528 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:04.663177 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:04.671438 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:04.688302 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:04.724550 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:04.731356 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:04.834095 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:04.860365 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:04.885999 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:04.920598 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:04.924937 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:04.954223 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:05.007455 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:05.022574 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:05.064699 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240503 08:02:05.074973 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:05.082032 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:05.092337 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:05.095023 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:05.095733 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:05.097020 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:05.098284 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:05.098976 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:05.131052 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:05.167742 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:05.168135 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:05.170442 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 08:02:05.171056 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:05.303463 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:05.308399 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:05.310532 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:05.315219 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:05.315495 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:05.316027 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:05.316699 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:05.324666 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:05.361300 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:05.424647 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:05.451196 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:05.466971 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:05.504118 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 84 similar messages]
I20240503 08:02:05.523106 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:05.526075 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:05.530934 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:05.531278 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:05.536026 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:05.537106 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:05.540022 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:05.561699 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:05.583954 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:05.589694 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:05.590319 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:05.590420 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:05.592401 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:05.616505 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:05.616869 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:05.626158 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:05.636770 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:05.710783 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 08:02:05.717895 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:05.724443 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:05.751020 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:05.755142 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:05.770758 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:05.773056 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:05.775070 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:05.782043 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:05.782691 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:05.819909 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:05.825304 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:05.828519 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:05.829399 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:05.840493 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:05.885583 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:05.976516 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:05.977695 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:05.986541 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:05.987488 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:05.992406 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:02:06.002156 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:06.012411 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:06.031903 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:06.046716 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:06.063745 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:02:06.088969 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:06.113407 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:06.171566 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:06.187996 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:06.194434 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:06.198819 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:06.200183 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:06.203184 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:06.235262 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:06.244182 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:06.320469 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:06.356622 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:06.379491 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:06.404757 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:06.452217 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:06.577100 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:06.621024 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:06.622663 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:06.630110 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:06.632854 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:06.644704 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:06.645413 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:06.667871 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:06.745457 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:06.745826 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:06.751771 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:06.770210 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:06.842370 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:06.844359 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:06.848001 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:06.851506 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:06.852237 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:06.859064 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:06.863778 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:06.864430 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:06.897315 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:07.003412 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240503 08:02:07.007371 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:07.026026 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:07.065366 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:07.067742 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:07.069625 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:07.080947 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:07.081926 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:07.083050 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:07.088443 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:07.099131 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.121394 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.127293 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.180619 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:07.273398 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:07.282104 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:07.283185 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:07.283448 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:07.287143 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:07.296622 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:07.301606 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:07.318310 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:07.344319 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.361698 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.412042 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:07.447793 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.487455 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.511529 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:07.513736 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:07.514612 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:07.518385 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:07.521214 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:07.522454 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:07.525734 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:07.526887 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:07.547400 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.571465 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.614745 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.672567 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:07.731998 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:07.732698 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:07.740660 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:07.741044 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:07.742533 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:07.743633 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:07.751035 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:07.771181 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:07.776345 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.828260 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:07.865070 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.867676 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:07.891767 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:07.980398 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:07.998709 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:08.021163 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:08.035423 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:08.051079 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:08.069898 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:08.096956 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:08.108815 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:08.114030 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:08.117251 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:08.142704 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:08.145241 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:08.151983 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:08.156731 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:08.164029 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:08.164013 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:08.169435 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:08.186885 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:08.248229 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:08.309070 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:08.340586 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:08.350996 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:08.351219 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:08.356505 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:08.363515 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:08.365082 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:08.365235 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:08.365701 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:08.369304 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:08.452482 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:08.476500 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:08.507982 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:08.510460 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:08.569988 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:08.570326 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:08.573343 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:08.573541 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:08.574433 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:08.592239 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:08.592521 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:08.606930 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:08.646920 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:08.706853 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:08.706848 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:08.794203 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:08.798558 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:08.800318 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:08.801296 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:08.802592 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:08.810151 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:08.813216 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:08.824285 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:08.896792 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:08.925112 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:08.959971 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:08.991135 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:09.020704 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:09.026999 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:09.027495 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:09.028442 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:09.034842 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:09.040602 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:09.054380 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:09.062053 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:09.137961 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:09.250766 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:09.251474 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:09.258621 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:09.267020 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:09.272719 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:09.283509 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:09.296478 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:09.307499 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:09.310366 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:09.352927 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:09.439193 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:09.484043 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:09.490995 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:09.609450 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:09.611240 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:09.652204 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:09.652479 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:09.668574 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:09.668948 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:09.672822 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:09.691521 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:09.703955 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:09.727553 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:09.735502 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:09.793536 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:09.806744 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:09.864289 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:09.870326 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:09.873385 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:09.875778 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:09.878536 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:09.887140 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:09.897740 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:09.898866 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:09.908267 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:09.923110 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:09.957152 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:09.987571 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:10.018821 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:10.023917 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:10.074272 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240503 08:02:10.089291 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:10.092993 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:10.094195 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:10.107650 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:10.109671 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:10.111713 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:10.112319 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:10.176559 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240503 08:02:10.193359 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:10.228297 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:10.246408 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:10.291664 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:10.297600 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:10.306313 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:10.316340 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:10.320187 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:10.321842 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:10.321899 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:10.330422 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:10.350239 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:10.367091 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:10.371582 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:10.371829 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:10.419608 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:10.473639 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:10.495487 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:10.501374 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:10.516764 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 08:02:10.537931 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:10.542546 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:10.546906 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:10.548316 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:10.550267 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:10.550493 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:10.551520 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:10.554623 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:10.557049 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:10.584215 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:10.592484 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:10.644847 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:10.683422 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:10.684130 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:10.716504 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:02:10.762310 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:10.767428 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:10.767575 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:10.773159 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:10.778337 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:10.787317 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:10.787431 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:10.792793 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:10.815016 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:10.830667 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:10.892033 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:10.902535 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:10.967438 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:11.005877 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:02:11.073688 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:02:11.074445 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:11.081264 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:11.128723 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:11.157352 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:11.170807 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:11.175736 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:11.184463 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:11.185182 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:11.185470 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:11.191630 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:11.194480 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:11.257938 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:11.258994 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:11.305758 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:11.373426 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:11.377135 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:11.381326 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:11.389283 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:11.390851 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:11.393302 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:11.411808 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:11.438026 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:11.458868 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:11.502830 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:11.503242 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:11.544448 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:11.609602 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:11.619271 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:11.619753 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:11.623871 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:11.627362 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:11.629706 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:11.648411 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:11.648775 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:11.774613 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:11.783982 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:11.836383 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:11.837026 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:11.850497 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:11.850809 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:11.851042 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:11.875377 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:11.875749 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:11.876296 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:11.882582 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:11.924667 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:11.958596 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:11.968385 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.019599 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:12.082566 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:12.083642 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:12.087590 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:12.092017 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 08:02:12.093808 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:12.099731 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:12.102986 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:12.118489 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:12.192621 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:12.274869 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:12.305104 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.311743 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:12.312199 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:12.312610 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:12.316272 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:12.322947 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:12.325122 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:12.325693 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:12.331223 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:12.339057 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.345095 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.345690 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.359668 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.432955 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:12.434006 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.439508 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.527518 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.535028 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.616436 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.714229 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:12.723481 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:12.725920 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:12.726320 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:12.732635 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:12.744902 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:12.751168 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:12.757637 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:12.818545 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.839412 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.873737 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:12.893949 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:12.915108 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:12.930250 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:12.931214 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:12.932024 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:12.933923 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:12.952307 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:12.953496 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:12.957108 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:12.957768 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:12.995642 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:13.002019 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:13.018880 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:13.027391 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:13.047909 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:13.053757 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:13.085039 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:13.112057 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:13.136065 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:13.141443 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:13.147428 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:13.150452 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:13.157969 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:13.158480 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:13.160427 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:13.169523 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:13.172940 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:13.220010 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:13.249507 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:13.364923 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:13.368664 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:13.371042 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:13.372326 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:13.372478 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:13.373198 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:13.383685 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:13.392297 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:13.415850 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:13.473404 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:13.505921 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:13.572258 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:13.596815 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:13.598923 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:13.599645 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:13.605895 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:13.607697 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:13.612845 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:13.614532 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:13.623911 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:13.653337 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:13.682953 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:13.688443 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:13.770102 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:13.811228 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:13.817229 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:13.818032 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:13.823603 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:13.824091 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:13.828552 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:13.838557 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:13.849341 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:13.973330 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:14.088079 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:14.104632 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:14.119127 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:14.134927 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:14.225322 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:14.228626 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:14.229912 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:14.235169 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:14.244079 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:14.245661 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:14.271993 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:14.273371 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:14.274451 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:14.274814 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:14.330243 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:14.335029 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:14.443425 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:14.450493 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:14.452874 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:14.454169 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:14.459419 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:14.460412 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:14.466548 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:14.467346 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:14.584944 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:14.589490 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:14.617363 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:14.635558 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:14.675182 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:14.676283 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:14.677786 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:14.683897 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:14.684983 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:14.688941 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:14.711165 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:14.762176 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:14.772985 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:14.774650 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:14.778131 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:14.799763 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:14.858726 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:14.866864 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:14.888971 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:14.892304 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:14.893615 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:14.900403 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:14.903321 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:14.903774 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:14.910620 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:14.937909 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:14.941197 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:14.947794 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:14.979501 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:15.050678 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:15.106096 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:15.111851 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:15.122007 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:15.122298 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:15.128657 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:15.138835 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:15.140470 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 08:02:15.141253 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:15.141803 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:15.143806 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:15.145320 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:15.178329 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:15.225677 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 08:02:15.311560 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:15.319679 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:15.321537 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:15.340263 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:15.342197 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:15.351244 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:15.357424 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:15.358500 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:15.366438 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:15.366523 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:15.367003 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:15.477926 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:15.516276 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:15.518422 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:15.524353 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:15.594828 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:15.608438 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240503 08:02:15.609114 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:15.617262 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:15.639456 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:15.656860 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:15.661386 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:15.663611 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:15.691372 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:15.721756 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240503 08:02:15.746402 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:15.776952 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:15.785804 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:15.786211 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:15.793033 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:15.801627 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:15.802706 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:15.802976 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:15.803625 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:15.889276 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:15.975293 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:15.975740 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:15.977958 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:15.990665 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:15.991318 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:15.993875 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:15.995038 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:16.009553 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:16.013072 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:16.096863 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:02:16.108134 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 08:02:16.109279 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:16.160411 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:16.193006 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:16.195560 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:16.197413 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:16.198100 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:16.211593 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:16.218889 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:16.222813 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:16.235777 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:16.236743 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:16.259094 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:16.350673 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:16.401551 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:16.404183 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:16.411844 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:16.418102 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:16.423863 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:16.448755 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:16.453842 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:16.525784 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:16.552363 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:16.609161 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:16.615751 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:16.617416 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:16.637282 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:16.638525 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:16.645524 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:16.648633 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:16.655401 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:16.672070 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:16.676471 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:16.696224 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:16.749099 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:16.837572 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:16.867041 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:16.876096 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:16.877709 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:16.880713 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:16.883096 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:16.890337 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:16.894838 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:16.918871 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:16.934125 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:17.042482 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:17.044229 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:17.052359 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:17.063588 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:17.114435 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:02:17.243256 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:17.283421 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:17.293229 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:17.308637 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:17.308944 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:17.312942 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:17.313295 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:17.315100 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:17.319711 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:17.329960 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:17.333431 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:17.338958 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:17.339509 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:17.351266 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:17.393572 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:17.433844 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:17.453280 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:17.517925 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:17.523404 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:17.526201 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:17.534155 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:17.534732 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:17.537158 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:17.537405 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:17.542006 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:17.557919 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:17.575059 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:17.584254 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:17.594734 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:17.628082 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:17.725363 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:17.748778 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:17.749349 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:17.753774 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:17.758481 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:17.760994 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:17.777683 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:17.809700 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:17.811749 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:17.856113 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:17.918601 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:17.935015 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:17.965224 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:17.965409 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:17.970283 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:17.971390 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:17.972116 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:17.979099 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:17.980144 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:17.996138 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:17.997843 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:18.037374 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:18.054008 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:18.109454 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:18.135773 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:18.164319 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:18.176595 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:18.183439 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:18.184686 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:18.189095 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:18.189399 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:18.206988 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:18.212442 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:18.240247 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:18.248804 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:18.251998 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:18.260376 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:18.349937 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:18.390290 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:18.398121 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:18.398584 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:18.401227 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:18.402252 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:18.402534 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:18.407029 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:18.415081 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:18.427348 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:18.434258 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:18.512367 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:18.542784 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:18.623921 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:18.633791 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:18.764721 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:18.818822 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:18.820263 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:18.823257 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:18.830530 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:18.836216 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:18.838687 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:18.852411 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:18.863123 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:18.960213 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:19.001111 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:19.030803 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:19.032089 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:19.034245 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:19.042482 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:19.043113 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:19.049499 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:19.074437 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:19.087126 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:19.137302 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:19.143074 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:19.163834 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:19.228991 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:19.230639 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:19.249994 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:19.261777 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:19.269554 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:19.273346 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:19.276538 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:19.276841 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:19.279300 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:19.300588 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:19.331054 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:19.450002 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:19.473517 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:19.480294 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:19.483728 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:19.503262 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:19.505481 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:19.518848 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:19.522825 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:19.533905 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:19.548239 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:19.593479 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:19.623613 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:19.641870 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:19.697501 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:19.698603 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:19.708664 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:19.708958 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:19.711833 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:19.712074 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:19.713193 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:19.721331 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:19.726508 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:19.729499 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:19.734745 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:19.790797 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:19.802857 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:19.882810 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:19.883919 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:19.896283 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:19.910729 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:19.913486 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:19.913583 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:19.933323 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:19.934018 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:19.934043 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:19.939596 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:20.029196 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:20.033628 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:20.035296 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:20.038938 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:20.042330 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:20.122123 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:20.130820 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:20.138288 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:20.152011 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:20.184942 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:02:20.189237 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:20.270471 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:20.287458 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:02:20.324219 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:20.336359 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:20.339263 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:20.344944 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:20.348552 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:20.349443 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:20.361011 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:20.385217 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:20.429687 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:20.445722 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:20.524490 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:20.549382 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:20.551996 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:20.552381 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:20.567875 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:20.568369 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:20.568850 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:20.583493 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:20.603269 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:20.717523 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:20.718199 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:20.728884 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:02:20.731941 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:20.735706 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240503 08:02:20.763991 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:20.768466 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:20.773240 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:20.782267 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:20.786124 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:20.787036 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:20.788686 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:20.791755 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:20.794726 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:20.801568 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:20.803498 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:20.806145 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:20.906455 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:20.981974 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:20.983551 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:20.984549 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:20.987320 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:20.994771 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:21.011785 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:21.029637 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:21.030295 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:21.033190 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:21.035122 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:21.038888 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:21.136858 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:21.139225 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:21.172335 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:02:21.201138 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:21.203121 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:21.204355 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:21.204603 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:21.213333 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:02:21.224261 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:21.230945 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:21.241708 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:21.302783 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:21.341029 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:21.399539 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:21.412683 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:21.415989 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:21.426774 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:21.427948 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:21.427990 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:21.433964 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:21.434111 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:21.456458 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:21.545496 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:21.564992 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:21.593190 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:21.610829 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:21.645254 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:21.661525 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:21.697906 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:21.837425 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:21.846081 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:21.853112 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:21.859107 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:21.862496 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:21.869374 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:21.897019 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:21.912078 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:21.919521 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:21.995057 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:22.026935 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:22.037740 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:22.060720 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:22.063211 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:22.069548 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:22.073727 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:22.084941 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:22.096156 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:22.098898 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:22.136353 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:22.142753 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:02:22.212199 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:22.221799 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:22.231539 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:22.277912 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:22.281358 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:22.283365 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:22.292286 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:22.302677 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:22.315558 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:22.318308 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:22.322655 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:22.341356 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:22.353341 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:22.386909 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:22.432257 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:22.434746 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:22.455590 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:22.480418 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:22.496925 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:22.498500 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:22.506940 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:22.511894 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:22.517132 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:22.518573 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:22.535136 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:22.535322 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:22.537111 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:22.551090 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:22.616127 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:22.647761 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:22.709025 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:22.713704 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:22.715320 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:22.717607 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:22.718300 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:22.727104 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:22.730039 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:22.731526 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:22.742760 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:22.743054 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:22.808763 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:22.910519 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:22.923152 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:22.926261 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:22.937887 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:22.938166 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:22.941466 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:22.943756 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:22.962440 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:22.985091 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:22.985622 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:22.999845 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:23.032433 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:23.099712 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:23.118000 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:23.118019 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:23.200425 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:23.215647 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:23.290587 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:23.312587 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:23.338485 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:23.343412 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:23.351893 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:23.353847 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:23.354878 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:23.367614 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:23.372570 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:23.396234 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:23.420993 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:23.447880 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:23.520062 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:23.546428 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:23.554126 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:23.565181 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:23.572057 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:23.574229 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:23.575174 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:23.591781 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:23.594058 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:23.615645 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:23.618533 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:23.641863 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:23.646030 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:23.686268 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:23.717628 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:23.785979 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:23.787746 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:23.799407 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:23.808847 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:23.809262 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:23.809314 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:23.814872 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:23.949326 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:23.996744 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:23.999577 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:24.004930 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:24.009568 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:24.014175 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:24.016985 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:24.018380 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:24.020403 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:24.021028 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:24.037226 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:24.065810 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:24.111951 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:24.123736 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:24.130136 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:24.223260 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:24.231019 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:24.231753 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:24.233114 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:24.243415 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:24.257886 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:24.260520 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:24.261519 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:24.454969 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:24.455173 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:24.457404 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:24.480777 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:24.481689 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:24.481766 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:24.492100 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:24.513893 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:24.514204 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:24.514288 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:24.515851 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:24.589069 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:24.609548 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:24.631927 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:24.726084 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:24.759768 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:24.775350 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:24.785696 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:24.877374 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:24.883297 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:24.887427 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:24.901983 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:24.902272 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:24.904206 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:24.908783 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:24.920670 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:24.928952 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:24.934860 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:24.936668 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:24.987035 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:24.992297 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:24.992782 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:25.040084 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:25.085016 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:25.116015 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:25.117092 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:25.119195 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:25.127856 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:25.128113 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:25.137492 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:25.142922 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:25.172820 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:25.190917 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 08:02:25.222607 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:25.229082 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:25.283991 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:25.315711 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240503 08:02:25.335537 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:25.335431 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:25.341964 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:25.347148 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:25.352950 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:25.357612 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:25.367597 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:25.373791 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:25.395565 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:25.414397 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:25.488809 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:25.557725 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:25.561259 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:25.561905 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:25.565724 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:25.566243 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:25.572028 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:25.575109 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:25.584709 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:25.590672 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:25.604089 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:25.604801 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:25.746188 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 97 similar messages]
I20240503 08:02:25.779286 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:25.786144 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:25.792160 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:25.793501 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:25.795486 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:25.797274 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:25.802825 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240503 08:02:25.817281 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:25.832654 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:25.844130 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:25.845029 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:25.847996 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:25.894379 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:25.896430 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:25.965828 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:25.990697 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:25.995391 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:26.003276 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:26.005789 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:26.016549 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:26.021355 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:26.025019 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:26.036424 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:26.043743 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:26.046998 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:26.141225 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:26.156725 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:26.216516 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:02:26.217255 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:26.217329 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:26.220273 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:26.229086 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:02:26.286258 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:26.286990 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:26.288151 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:26.370437 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:26.408807 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:26.409165 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:26.411922 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:26.413409 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:26.418295 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:26.420060 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:26.432801 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:26.572088 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:26.618925 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:26.621409 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:26.628506 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:26.628799 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:26.630209 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:26.630334 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:26.634203 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:26.639283 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:26.640345 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:26.641782 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:26.642146 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:26.651182 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:26.745054 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:26.804865 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:02:26.849040 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:26.852517 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:26.852679 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:26.857988 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:26.880630 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:26.880658 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:26.904557 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:27.101589 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:27.102566 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:27.102977 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:27.103271 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:27.117682 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:27.117707 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:27.119568 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:27.122265 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:27.136345 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:27.138180 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:27.146082 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:27.162571 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:27.173563 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:02:27.174301 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:27.194401 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:27.233752 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:27.234380 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:27.276402 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:27.285429 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:27.317936 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:27.329948 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:27.331912 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:27.337224 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:27.337224 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:27.337875 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:27.339664 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:27.343503 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:27.384191 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:27.436898 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:27.468672 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:27.543638 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:27.548727 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:27.551229 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:27.552407 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:27.554762 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:27.556254 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:27.558543 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:27.567777 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:27.601572 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:27.618913 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:27.663352 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:27.679574 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:27.682556 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:27.695518 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:27.736371 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:27.741890 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:27.749624 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:27.798861 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:27.837399 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:27.939114 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:27.948976 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:27.950547 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:27.951328 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:27.953936 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:27.954707 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:27.985539 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:28.008425 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:28.107899 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:28.129572 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:28.154820 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:28.161187 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:28.163489 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:28.165200 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:28.174989 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:28.179489 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:28.183995 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:28.198239 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:28.245793 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:28.270761 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:28.301014 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:28.363768 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:28.367107 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:28.370962 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:28.375306 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:28.376235 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:28.390697 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:28.390982 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:28.396646 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:28.417464 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:28.519630 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:28.538444 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:28.590860 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:28.594198 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:28.595264 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:28.601110 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:28.602141 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:28.610877 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:28.611393 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:28.640235 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:28.646281 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:28.654183 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:28.683018 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:28.685387 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:28.690110 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:28.753774 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:28.777540 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:28.786438 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:28.787039 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:28.818240 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:28.819564 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:28.824164 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:28.824666 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:28.829460 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:28.832137 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:28.840665 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:28.845749 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:28.992859 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:29.021541 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:29.031352 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:29.039256 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:29.039250 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:29.044451 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:29.050529 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:29.070928 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:29.075378 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:29.106335 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:29.130405 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:29.146840 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:29.184010 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:29.193259 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:29.215654 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:29.294129 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:29.445279 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:29.447345 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:29.448103 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:29.455554 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:29.458513 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:29.460587 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:29.467458 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:29.549662 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:29.570302 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:29.590217 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:29.594828 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:29.606384 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:29.625913 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:29.646597 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:29.648021 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:29.651253 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:29.655768 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:29.668797 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:29.671765 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:29.672397 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:29.676131 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:29.688647 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:29.694675 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:29.775571 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:29.818830 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:29.862596 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:29.863756 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:29.871694 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:29.873111 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:29.874019 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:29.876809 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:29.877292 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:29.921671 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:29.926779 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:29.931046 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:29.941964 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:29.954828 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:30.005378 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:30.070370 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:30.090211 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:30.095278 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:30.098932 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:02:30.099548 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:30.100648 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:30.115967 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:30.116119 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:30.133646 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:30.135900 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:30.154040 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:30.195989 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:02:30.206537 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:30.217592 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:30.224125 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:30.304281 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:30.319303 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:30.323872 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:30.325143 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:30.332345 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:30.334640 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:30.334682 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:30.350737 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:30.372879 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 102 similar messages]
I20240503 08:02:30.375797 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:30.503367 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:30.539294 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:30.540755 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:30.548537 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:30.559995 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:30.574666 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:30.577342 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:30.578548 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:30.600891 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:30.606388 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:30.628362 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:30.655608 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:30.675807 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:30.685488 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:30.730748 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:30.733836 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:30.768476 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240503 08:02:30.805065 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240503 08:02:30.872779 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:30.906174 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:30.936828 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:30.939433 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:30.943468 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:30.950762 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:30.953732 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:30.965965 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:30.970533 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:30.978698 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:31.005072 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:31.140429 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:31.149380 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:31.150249 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:31.152216 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:31.167068 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:31.169003 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:31.174530 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:31.185017 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:31.229187 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:02:31.230850 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240503 08:02:31.242869 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:31.254248 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:31.291610 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:31.294533 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:31.294708 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:31.295955 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:31.321588 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:31.335415 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:31.368278 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:31.368492 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:31.372552 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:31.374186 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:31.375900 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:31.388797 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:31.388792 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:31.416924 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:31.558605 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:31.578672 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:31.582631 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:31.582762 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:31.586143 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:31.599254 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:31.600833 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:31.612943 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:31.621568 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:31.638118 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:31.721861 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:31.725911 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:31.794948 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:31.795864 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:31.800379 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:31.803333 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:31.804085 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:31.806185 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:31.820636 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:31.831898 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:31.832610 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:31.839648 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:31.880174 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:31.906116 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:31.937811 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:31.994226 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:32.016281 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:32.018227 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:32.023382 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:32.023658 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:32.028805 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:32.044013 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:32.057406 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:32.064364 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:32.064941 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:32.100244 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:32.125412 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:32.169981 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:32.224731 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:32.279042 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:02:32.282955 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:32.336900 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:32.346621 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:32.390296 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:32.433965 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:32.435829 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:32.441191 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:32.446521 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:32.446996 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:32.456821 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:32.456851 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:32.462755 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:32.479956 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:32.507684 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:32.513720 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:02:32.524677 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:32.619978 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:32.651561 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:32.652468 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:32.660403 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:32.665904 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:32.674161 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:32.676002 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:32.692566 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:32.710646 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:32.741123 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:32.772181 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:32.801789 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:32.809021 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:32.866518 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:32.871742 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:32.873481 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:32.874023 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:32.874248 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:32.874287 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:32.880136 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:32.887490 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:32.907572 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:33.061622 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:33.084542 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:33.084864 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:33.087980 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:33.098070 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:33.098557 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:33.105156 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:33.121981 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:33.125272 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:02:33.139766 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:33.143640 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:33.160717 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:33.177871 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:33.186934 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:33.212527 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:33.254591 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:33.324596 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:33.332177 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:33.343545 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:33.344787 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:33.348995 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:33.357630 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:33.364538 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:33.370999 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:33.380870 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:33.432720 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:33.490763 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:33.552796 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:33.565462 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:33.565726 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:33.566138 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:33.567214 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:33.569099 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:33.571877 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:33.573130 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:33.586896 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:33.698199 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:33.708848 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:33.789197 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:33.815343 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:33.862408 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:33.874627 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:33.962528 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:33.962801 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:33.965536 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:33.974673 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:33.976194 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:33.977874 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:33.983196 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:33.988605 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:34.001937 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:34.005386 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:34.011571 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:34.031970 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:34.055162 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:34.131212 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:34.179646 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:34.187816 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:34.193199 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:34.198524 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:34.214743 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:34.215778 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:34.225661 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:34.238811 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:34.256129 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:34.310755 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:34.325943 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:34.395725 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:34.399881 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:34.400835 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:34.404964 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:34.406562 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:34.418150 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:34.424685 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:34.441020 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:34.445991 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:34.456465 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:34.544359 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:34.606897 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:34.614212 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:34.622660 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:34.624157 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:34.624444 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:34.624859 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:34.627017 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:02:34.635445 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:34.648209 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:34.713593 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:34.748477 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:34.755843 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:34.768644 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:34.823148 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:34.833981 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:34.838132 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:34.838706 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:34.847858 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:34.855362 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:34.861208 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:34.885272 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:34.925143 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:34.953522 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:34.978834 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:34.985006 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:35.060287 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:35.063606 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:35.063850 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:35.069531 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:35.071130 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:35.073822 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:35.076198 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:35.077286 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:35.090560 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:35.162868 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:35.193827 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:35.204053 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:35.237003 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:35.265556 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:35.283481 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 08:02:35.294590 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:35.373411 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:02:35.405246 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:35.483203 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:35.483894 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:35.490031 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:35.505977 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:35.506633 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:35.508329 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:35.508818 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:35.514824 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:35.518509 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:35.523519 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:35.573220 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:35.594180 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:35.594810 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:35.608055 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:35.653623 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:35.708128 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:35.711918 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:35.718097 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:35.719341 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:35.724074 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:35.724383 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:35.724589 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:35.755084 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:35.806423 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 08:02:35.807080 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:35.818483 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:02:35.847025 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240503 08:02:35.921610 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:35.924068 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:35.928508 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:35.932595 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:35.936147 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:35.939178 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:35.942765 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:35.952323 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:35.958966 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:36.081550 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:36.086136 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:36.098950 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:36.144093 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:36.147115 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:36.151818 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:36.154371 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:36.165627 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:36.167001 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:36.178063 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:36.194939 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.233970 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:02:36.234944 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:36.242614 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:36.254935 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.263104 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:02:36.275367 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.303050 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.321019 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:36.358373 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:36.359938 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:36.365909 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:36.373334 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:36.375838 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:36.378876 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:36.392366 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:36.406541 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:36.419584 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.484320 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.533887 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.576987 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:36.578887 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:36.582317 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:36.589962 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:36.592815 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:36.600205 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:36.604094 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:36.630931 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:36.709795 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.712077 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.734997 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.772028 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:36.819783 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.823470 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.826943 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:36.847033 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:36.978480 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:36.993337 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:36.995983 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:37.000774 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:37.001039 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:37.005717 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:37.016976 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:37.022217 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:37.033810 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:37.036649 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:37.051900 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:37.069088 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:37.225744 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:37.235670 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:37.237272 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:37.249631 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:37.250859 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:37.260495 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:37.264948 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:37.265074 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:37.279666 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:37.279773 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:37.297513 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:37.298698 6823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:02:37.450372 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:37.459347 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:37.467006 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:37.474529 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:37.478966 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:37.482538 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:37.483117 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:37.504122 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:37.511193 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:37.527940 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:02:37.611253 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:37.688556 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:37.694456 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:37.696452 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:37.698786 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:37.711254 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:37.714792 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:02:37.715874 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:37.720310 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:37.728677 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:37.737723 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:37.806381 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:37.854158 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:37.908111 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:37.908375 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:37.910492 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:37.911628 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:37.914471 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:37.915594 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:37.916949 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:37.922003 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:37.949296 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:37.959533 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:37.963408 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:37.998296 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:38.002279 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:38.077838 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:38.103091 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:38.125677 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:38.128787 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:38.129801 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:38.130724 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:38.148994 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:38.149273 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:38.168201 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:38.172585 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:38.195420 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:38.205351 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:38.298377 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:38.356353 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:38.376876 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:38.461352 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:38.484441 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:02:38.539587 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:38.544659 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:38.545282 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:38.545967 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:02:38.551826 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:02:38.566627 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:38.567904 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:38.575064 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:38.575140 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:38.575390 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:38.608876 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:38.659965 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:38.662309 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:38.685833 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:02:38.754645 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 08:02:38.754976 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:38.756192 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:38.760710 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:38.761211 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:38.779575 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:38.781962 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:38.786886 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:38.796155 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:38.797345 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:38.824152 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:38.916421 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:38.920279 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
I20240503 08:02:38.964030 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:38.964105 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:38.969717 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:38.978227 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:38.979387 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:38.995108 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:39.005036 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:39.013854 9701 consensus_peers.cc:485] T ce60559e643f453cb099d1d39b0175c9 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:39.068081 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:39.137102 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:39.157248 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:02:39.171721 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:39.179598 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:39.181672 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:39.195226 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:39.198540 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:39.201661 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:39.218640 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:39.221534 6823 consensus_peers.cc:485] T fe4d550eda29472d8c9ae494d25efc83 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:39.228977 6954 consensus_peers.cc:485] T bc3d64c714fd471b9f055af7cb172a39 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:39.268833 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:39.315032 9439 consensus_peers.cc:485] T 9cfe98aefcfe4380bb64f14df1a24318 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:39.328140 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:39.343324 7085 consensus_peers.cc:485] T ad1a17fdefd3425aa0cda9a8fdc827c0 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:39.375460 6954 consensus_peers.cc:485] T 4ef6defd02954e009565bae98f5e8418 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:39.398975 6954 consensus_peers.cc:485] T be911252c4da43708b7ce6592d9b17f6 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:02:39.399330 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:39.403049 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:39.410606 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:39.415513 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:39.419435 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:39.423049 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:39.432046 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:39.436101 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:39.533145 9701 consensus_peers.cc:485] T fbc7505d3ef6416c9d505bfe899f73ed P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:39.608809 9701 consensus_peers.cc:485] T fe38557414dd4f509604884bb23cc8ee P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:02:39.627967 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:39.629037 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:39.630542 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:39.633719 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:39.635890 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:39.636237 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:39.675127 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:39.676940 7085 consensus_peers.cc:485] T 866c1bf17d444315bb3eb83a7a283b54 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:39.780465 9701 consensus_peers.cc:485] T dae94f108f8142379c271e9fc22524ef P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:39.903820 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:39.904415 7216 consensus_peers.cc:485] T 795f69249afe444ba0828875e1839b8c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:39.936693 7216 consensus_peers.cc:485] T 721c34a8b6a348e5890d462ba7da938c P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:39.999760 7216 consensus_peers.cc:485] T 6e63db30c0454722b6a794a9259a67c0 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:40.006775 9570 consensus_peers.cc:485] T fef7df80245c4b2c9fd3efd8331ba9e4 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:40.007517 9439 consensus_peers.cc:485] T 4f2c21fc7842455f8617cb0405becbcf P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:40.063489 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:40.070850 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:40.077958 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:40.080868 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:40.085278 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:40.085827 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:40.086247 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:40.101421 6823 consensus_peers.cc:485] T 61737b6ad38c470b8b7d6fceba5dbbad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:02:40.131052 9701 consensus_peers.cc:485] T 7009acabeee34602a3ce7462ed5a2d8a P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:40.160549 9439 consensus_peers.cc:485] T 5388c1fa25bc4f19a2aef9d8960e88aa P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:02:40.212152 6954 consensus_peers.cc:485] T 7d3eb284c0fc48b4912168c434844284 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:40.212812 7216 consensus_peers.cc:485] T 16a4b20f1cc24ed09fdaaa2f9764e709 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:02:40.283706 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:40.292301 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:40.294286 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:40.298552 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:40.298818 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:40.308267 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:40.310099 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:02:40.319101 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 08:02:40.321285 9570 consensus_peers.cc:485] T c2b5e8db1eec44a7881dde976e8a0a12 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:40.327112 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:40.344437 9570 consensus_peers.cc:485] T cf2beba18ec6449c90c50e200ff40d68 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:40.403898 6954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 08:02:40.404582 6954 consensus_peers.cc:485] T fb1bff0a8228454f95c14762536a4c82 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:40.433343 9439 consensus_peers.cc:485] T 100a1be1cc104b8cb0412e764461f25b P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:40.440413 9701 consensus_peers.cc:485] T 3e7d7f7ee4384a2ba8bd0dac0ddad757 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:40.488423 7085 consensus_peers.cc:485] T 7c2386cd009847d9a1fe7a959f47086c P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:40.516839 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:40.523864 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:40.529875 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:40.537570 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:40.541249 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:40.550941 7085 consensus_peers.cc:485] T 61b93171848746be89447d2ef90459a7 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:40.557102 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:40.557698 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:40.609932 9439 consensus_peers.cc:485] T df569029c1514bc5ae4d77327cd03478 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:40.634918 9701 consensus_peers.cc:485] T 75f59b9fe4a54e7d8beabce7589800b1 P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:40.651382 7085 consensus_peers.cc:485] T 5390e70b0449449aa3af05112bd29797 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:02:40.741355 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:40.744112 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:40.746474 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:40.759296 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:40.770438 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:40.776477 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:40.787163 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:02:40.829962 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:40.830471 7085 consensus_peers.cc:485] T 1c23c1c8b02147b780c9e1aebc48f24a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:40.843819 6954 consensus_peers.cc:485] T abb55c09be324f37a1a546e73008a80d P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:40.851387 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240503 08:02:40.890242 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 08:02:40.909564 9701 consensus_peers.cc:485] T 0aa18637596d4a39aff9a4e4a261228d P a372e98a670f49e9adca46ab5c836c4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:40.914973 9570 consensus_peers.cc:485] T faee1f8d939e4487a014044772ba2daf P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:02:40.952247 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:40.956615 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:40.957103 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:40.957952 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:40.969341 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:40.969923 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:40.973554 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:40.997876 7085 consensus_peers.cc:485] T 5365969434f142b8a5757a9a73bc109a P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:41.029439 6823 consensus_peers.cc:485] T bed0aa049b8b4e7c926d6417acf4f068 P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:41.032100 6823 consensus_peers.cc:485] T 5293a26b65484ae5a9e76469e644c5bc P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:41.032591 9570 consensus_peers.cc:485] T 27139acdbdb54c248f35fcf9ec023f4d P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:41.062278 6954 consensus_peers.cc:485] T cc2d83ce05384dc9b3e306569fa200b3 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:41.110460 9570 consensus_peers.cc:485] T cab217306da64dd7bb123e7fb890d103 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:41.129856 6954 consensus_peers.cc:485] T 5fa65337d4dc48df91409554d00f5f17 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:41.170130 6954 consensus_peers.cc:485] T 25979eb7a4fe48939d0585bf27468918 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:41.172312 9570 consensus_peers.cc:485] T dc3367cc41234d52b74eee94d6e67069 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:02:41.172814 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:41.186506 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:02:41.186602 7261 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:41.193343 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:41.196213 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:41.198098 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:41.232435 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:02:41.281324 7085 consensus_peers.cc:485] T 96289b8f77bb4241bade14b1011e1de4 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:41.299543 9439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:02:41.357281 7216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:02:41.357930 7216 consensus_peers.cc:462] T dd6aa70d9c4c4baea597bbb9788fdb37 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111)
W20240503 08:02:41.409121 7216 consensus_peers.cc:485] T d66715134ff740129267a5300af8cb19 P 5227b26243484bbfb177a995ceb50afc -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:02:41.476852 9570 consensus_peers.cc:485] T 1ac0bb7b2c95434aa38f0f0e3718e89c P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:41.477421 7085 consensus_peers.cc:485] T ad4d427d99214106ba9ead0ec30b4687 P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:41.507949 7085 consensus_peers.cc:485] T 6bbf3e180fcf4d7ab56f7b7e967ab7bc P 00a3f8e159c64327832d79fb32cdcca8 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:41.512768 9439 consensus_peers.cc:485] T 6e051a416bb446df91e90eba13206341 P cb73994515af4880b05383e78368bb4a -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:41.585604 6954 consensus_peers.cc:485] T 47ae0cf462874f26985099f1c5132ae0 P 9c085a3241f14cb7ba1332b508f4b57d -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:02:41.613507 7129 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:02:41.614612 6998 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:02:41.625022 6868 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:02:41.627810 9484 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:02:41.628219 9615 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 08:02:41.629750 9747 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:02:41.655222 6823 consensus_peers.cc:485] T 21f55eb986bd4b7bbda10e683f3bfcad P c4218904dd7e42d2be95b901a6e5db92 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn't send request to peer 3befbb2d206f4be89d3e07f38837374b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37815: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:02:41.656559 9570 consensus_peers.cc:485] T 659657d85679433abdcbe711e4741b67 P 159177b3caeb434baa98e04136d050b2 -> Peer 3befbb2d206f4be89d3e07f38837374b (127.0.99.133:37815): Couldn'